Add MediaRecorderOptions.forceVideoFrameRate (#113) - #114

ping @jan-ivar

I am concerned about what happens when input frame rate varies or takes a dip, since a VideoStreamTrack by definition doesn't give you a constant frame rate.

I think a related problem that is harder to solve is offline video recording (ref OfflineAudioContext), where an application could render as fast as possible but you'd still get a deterministic result from the recorder.

I think such a solution could be generic enough (supporting offline/non-offline) to be re-used for the use case you have here.

I guess I'm thinking of something like AudioWorklet but for timestamps/frames/audiobuffers. This would allow for the application to decide what to do when there are framerate changes.

@miguelao I share @Pehrsons's concerns here. Can you elaborate on the use case you want covered?

Note that for finer control of frame rate, you could render your source in a captured canvas. Probably not applicable, but I found it interesting how MediaRecorder plays well with a controlled output of frames.

The use case here would be to force a webm timestamp that is not the wall clock time,
which is what MediaRecorder uses now (at least in Chrome). Even if the input to MR came redirected via a , this restriction would not be lifted, for example:

a) we have an input camera capturing at, e.g., 120fps, but want the recording at 30 fps as an slow motion output.
b) we have an input camera capturing at e.g. 0.5fps but would like the recording at 30 fps as an example of time lapse.
c) we have a bursty input, e.g. a tab capture, and would like to smooth out the bursts.

Surely we could leave this as a post processing step, by retimestamping the produced multiplexed container, or provide a Javascript callback to query for nextTimestamp(), but the current option proposal seems to be to cover enough ground with a tiny addition to be worthwhile.

wdyt? Alternatively, @Pehrsons could you elaborate on your proposal?

What I don't understand is how this would help with c). It wouldn't smooth out the bursts. Yes, if you look at timestamps in the produced video track they would have a perfect, fixed rate. But the perceived frame rate would be much lower during a burst (e.g., 120 fps turned to 30 fps) than during normal operation (e.g., 60 fps turned to 30 fps), and IMHO that is not (always, perhaps) something the application would want.

If you think of a function that takes wallclockTimestamp as input and returns the timestamp that should be used in the muxer it would be up to the application to decide how to handle high framerate, low framerate, variations in framerate, bursts, and so on.

Re-using it in a larger context of offline recording is perhaps a bit naive, but I like the idea of giving power to the application to handle use cases we don't understand yet.

I've taken a look at #113 and this, and both the slo-mo and time-lapse cases seem like they would be better served by a scale. In both those cases, if the input frame rate is not constant (jittery timing or real changes of recording rate) is it not better to preserve that?

And then there is the case where the input frame rate is variable but we want the output to have a fixed frame rate without changing the overall duration of the video. For that I think this option makes sense. Solving just that problem first also seems fine to me.

@miguelao

I agree with @foolip that slo-mo and timelapse (a and b) are better served by a scale feature.

I agree with @Pehrsons that for "smoothing bursty input" (c) the result seems undesirable, because it would effectively slow down and speed up in places of burts. Imagine music being attached to this, it would probably sound pretty awful, like off an old VHS tape.

As @Pehrsons illustrated for me, if the camera was filming an actual wall clock we'd at least expect it to move at a consistent speed in the output, whatever frame rate it's showing.

Oh, wait, was the proposal to take every frame and give it a constant duration, so that all frames are in the output? I didn't get that in #114 (comment), I assumed some dropping of frames to limit the frame rate.

@foolip the proposal was to ignore the wall clock timestamps in the multiplexed container creation, and use an artificial monotonic source of a certain, user-defined, speed. Nothing about limit frame rate.

I think the consensus is more about having a scale Options dictionary entry which would serve all cases, correct? (+1 this comment if you're ok with it).

I think scaling solves some use cases but far from all (including ones we haven't thought of), which is why I still prefer a solution where the application can decide on a per-frame basis.

So, I am fine with anything y'all want to implement, and will now get out of the way :)

@Pehrsons, @jan-ivar, how would such a user-provided-callback look like spec-wise? The input parameter could be a DOMHighResTimestamp representing the wall clock time, should it also received a Track type and perhaps an ID..? Personally I'd rather not bake too much in the Spec without any particular goal, so having a frameRateScale parameter at this point would do it for me.

Ping @jan-ivar @Pehrsons

I'm afraid I don't have the experience or time to tell you how that could be specified, but I'd start looking at AudioWorklet in WebAudio if I were to figure it out.

I think a particular (larger) goal is just what this spec needs. It's been drifting for too long.

@Pehrsons I took a look at AudioWorkletNode(which looks cool btw) and discussed offline with @alvestrand and I think that we should proceed with the simple clock scale options parameter and leave more complex use cases, legit and interesting as they are, to other specs such as e.g. mediacapture-worker or Streams, where the timestamps can be modified in creative ways.

The current proposal is a small solution to a small problem that appears internally to the Media Recorder (between the encoders and the muxer, concretely). @jan-ivar WDYT?

Reading the thread, it seems that we have consensus that "forceVideoFrameRate" is the wrong solution to the problem stated, and that if looking for simple solutions, "clockScalingFactor" or some name like that (default 1.0, range 0+epsilon to +high) is an adequate tool for the time-lapse and slo-mo video cases.

To be honest, I'm really not sure this should even be a part of MediaRecorder. Maybe I'm alone in this view, but I think MediaRecorder should primarily be about encoding captured media (that's really the pain point for javascript). Stuff like frame rate options should really be a part of getUserMedia, which is primarily about video capture.

Possible I'm being pedantic.

@jnoring the discussion here is about the relation between Video/Audio Frame timestamps and the recorded chunks' presentation timestamp that go into the multiplexed file (webm/mkv in Cr case), that currently just follow the wall clock time. The framerate of the data source can and should be configured via getUserMedia(). This would allow webm timestamping at e.g. 30fps while having a video source being configured to produce 120fps. Makes sense?

I don't understand the timelapse and slo-mo use cases here. For small-scale factors, these seem like playback properties to me, not recording properties. In other words, the desired effect seems achievable by either speeding up playback or slowing it down.

For actual frame-skipping timelapse, a perfectly performant solution to me seems to be to render a frame into a canvas that's hooked up to a recorder with no framerate specified. This is defined to generate a new frame every time the canvas changes. If I do this at e.g. a 1 second interval, then it'll throw away all other frames from the camera, producing a true timelapse recording without extraneous frames.

@jan-ivar, the slo-mo or time-lapse can be achieved in either way, e.g. slowing down/speeding up the playback, or modifying the timestamps in the container. The first method, however, needs to communicate to all users that the video speed is a particular one, and how would you guarantee that?

In the canvas case, the current scheme of things would record a burst as, well, a burst of frames. If the user/dev wants to address this problem, they'd need to use the MediaStream/Track constraints. The current proposal would simply speed up/slow down the produced file.

@miguelao Again, I don't understand who the "user" is for this. I understand that lying about the frame rate creates a certain effect, and that some web developers may have viewers in mind for such a video effect. But mediaRecorder seems like the wrong place for this lone video effect, a single-knob hack, and I don't find it compelling for that reason.

For the case where the user is the consumer of a produced file, maybe uploaded to a server, there must lots of video editors that can edit the file and change its frame rate, as well as do all kinds of other effects in the process before the recording is eventually staged for viewers to view through media players outside the control of the wbe developer.

For the case where the user is the immediate viewer of a file recorded locally in the same client, the JavaScript should be able to control playback speed.

Also, what would happen to audio?

@miguelao Hmm, my canvas capture timelapse demo didn't work the way I expected it to (Requires Firefox Beta 52 due to me using async/await. Doesn't work in Chrome for some reason).

For some reason I thought updating canvas twice a second would produce something that played very fast, i.e. timelapse - I don't even know what framerate I expected - but instead it plays back at 2 fps. Maybe i can be convinced. ;-)

To clarify where I stand now, I won't argue strongly against a scaling feature - as somewhat of a complement to canvasCapture driving creation of frames - other than to say I'm confident we'll come up with something better if we address offline recording first.

I see two kinds of offline recording: The first is recording frames faster than playback, typically as fast as possible. The second is frame generation from some time-consuming effort, whether it be timelapse or rendering some scene. In the second case, what if the time it takes to generate each frame varies? For that reason, I suspect @Pehrsons is on to something when he talks about per-frame control. But I don't think we'll know what that looks like until we've discussed offline recording.

Re. offline recording: what about a scenario in which a

ping @jan-ivar

I think we agreed in conversation that if this feature should exist, it should be a timestamp multiplier, with a name something like scaleTimestampBy.

An important question not answered above - "what about audio?"
Other solutions that do time compression either drop sound altogether or do spectrum shifting - the spec needs to specify which.

I worry two things with the same frame rate that aren't otherwise coordinating, won't necessarily keep in sync. Could this cause occasional frame skips? In any case, this only solves one narrow use case. I would hold off on this until we've tackled offline recording for a majority of use cases. I think we'll be more likely to get this right at that time.

Re @miguelao's offline idea: how would you solve a case where the application is generating the content? For instance through a canvas, preferably off the main thread.

We should IMHO also integrate with AudioNodes from an OfflineAudioContext. That's already a well defined way to do offline audio.

Getting off the main thread requires workers, and we don't have that yet.
Neither defining nor implementing MediaStreams with workers is entirely trivial - many have expressed interest, but nobody's signed up to do the work yet.

Getting off the realtime clock also requires significant model work. I think this needs to be a different spec than the ones we're working on now.

Answering a few questions:

• This PR addresses a small use case, which is the modification of the multiplexer timestamps to follow the wall clock times a multiplication factor (proposed name:scaleTimestampBy).
• As it stands right now, what happens to audio is TBD, I think it's best to agree first on: scaling factor yes/no, before discussing it.
• Albeit very interesting, this PR does not address offline recording cases nor a per-(possibly recorded)-frame manipulation API. We can discuss it in a(nother) Spec issue, wdyt?

I agree with jib when he says

In any case, this only solves one narrow use case. I would hold off on this until we've tackled offline recording for a majority of use cases. I think we'll be more likely to get this right at that time.

Seeing that the argument of solving one small use case doesn't convince you @jan-ivar and @Pehrsons, I'm eager to work with you guys on your proposal for offline recording, be that here or elsewhere (wicg?). At any rate, it would be beyond the MediaRecorder 1.0 Spec.

Closing this PR and associated issue.