Hi,
I’m trying to wrap my head around the rolling shutter effect, specifically why it happens.
I’m having a hard time understanding how the readout speed affects the image. If I understood correclty, when in electronic shutter mode the pixels are exposed as indicated by the shutter speed (e.g. at 1/1000 each pixel is exposed for 1/1000 of a second).
If the readout takes 1/100 s to scan the entire sensor, what happens exactly when I take the picture? Do the pixels start firing sequentially as the shutter speed dictates (i.e. 1/1000 s each, sequentially)? If that is the case, do they wait for the readout to catch up or do they continue firing? If the latter, by the time the readout reaches the second pixel, the eleventh pixel is firing, so there are 10 pixel between the one firing and the one being read. Does it work like this?
If the pixels are exposed for 1/1000 s and then turned off and their value stored, wouldn’t that mean that the image should not be affected? I mean, they saw the subject for 1/1000 s and the motion should be frozen, they are just waiting for the value to be read. Just like if you asked 10 people to open their eyes for 1 second (shutter speed), one after the other, and draw what they see. They saw if for one second each, so at most the difference in the position of what they saw should cover 10 seconds. Then they can take hours to draw what they saw (readout speed), but what they saw specifically wouldn’t be afftected by how long it takes them to draw it. Am I wrong here maybe?
Also, in general, why is mechanical shutter not as affected (if affected at all) by the rolling shutter effect? Does the sensor capture light differently when in mechanical shutter mode?
I just don’t get it. I feel like I’m close to understanding why, but I still don’t.
I know I’m probably weird for focusing so much on something technical like this, but it just bugs me so much.
Any help is greatly appreciated, really.
I could try but honestly Destin at Smarter Everyday will do such a better job than me. I also included a slow mo guy vid on the same subject. It has been a bit since I watched either of these vids so I hope they answer your question.
Thanks for the videos, I love both of those channels! But they don’t really explain the intricacies behind the rolling shutter, so they don’t answer my specific questions :(
Another good video from Captain Dissillusion:
This is a great video! You made me discover such a good channel. However, it gets close but not enough :(
Try holding your hands a few cm/in in front of your face. Fork your fingers to make a narrow slit you can see through. If you move your hands from top to bottom you will be scanning only part of the scene at a time. So if something is moving (car, child, etc.) it will be in a different place when your “slit” is at the bottom than when it was at the top.
Most camera sensors can only be read one line at a time. This means that the second line isn’t captured until the first line is completed. This goes for all of the remaining lines on the sensor of which there are thousands. For example a 24 megapixel camera might have around 4000 lines of pixels to capture.
The image processor captures the state of the top line and moves on to the next. By the time it’s captured the 2nd line, if the subject has moved, that line is no longer aligned with what was captured in the first line. Continue this 3998 more times and you have a frame of a video where the bottom of the frame is further ahead than the top of the frame.
To slow this down to a human understandable scale, while watching a video close one eye, then open it and close the other. By the time you’ve opened the second eye and closed the first things have moved. Just don’t do it while driving.
The basics: the photo sites on a camera sensor can be though of as buckets with a color filter on top of them, so each site captures only red, green, or blue light. When you take a picture the site is emptied, filled for the exposure duration, and then measured. The fuller the bucket the more intense the light was at that spot on the sensor. Buckets have a finite capacity, so if you fill them too much you’ll reach “full” and blow the highlights.
Electronic shutters fall into two camps: progressive (also called rolling) and global.
On a global shutter sensor, all photo sites are emptied, filled, and read simultaneously.
On a progressive shutter, this happens row by row so you empty, fill, and measure the first row, then the next, then the next. Even if each photo site is only exposed for 1/10000 of a second, the process of reading all the photo sites can take a while so the “time the picture was taken” progressivly gets delayed as you go across the sensor. On Sony’s A7R V this process takes about 1/10 a second. Even if your exposure is set to 1/10000 of a second, the process of taking the photo still takes 1/10 second. If anything was moving at a decent clip during the exposure it will look stretched, because the photo is essentially a series of row-by-row photographs that get stacked together over that 1/10 a second. Faster sequential sensors, like the Nikon Z9, can achieve 1/270 readout speeds which minimize, but don’t eliminate, this effect.
It’s worth saying that mechanical shutters are also progressive shutters and that a full frame mechanical shutter takes somewhere between 1/200 to 1/250 to traverse the sensor. This is why the Z9 doesn’t have a mechanical shutter - there isn’t a ton to be gained from having one. Mechanical shutters are significantly faster than the electronic shutters on most cameras, but it’s still possible to get the rolling shutter effect with a mechanical shutter if you’re planning quickly or taking a photograph of something moving fast enough.
The only way to eliminate the effect completely is with a global shutter.
Thanks for getting into the details.
I’m still not 100% there…
I still am not sure how the exposure is started: is it row by row or is it the entire sensor and it’s just the readout that is sequential? In the first case, then it would make sense to me because they would gather 1/1000 s of light and then, even though the readout is slow, they are physicall not receiving any more light because the shutter is closed and it’s dark, so the image is not affected. If the exposure starts sequentially as well at the readout speed, I would not understand how it could keep up with the curtains, because the curtains would fly over the sensor while the pixels didn’t have the chance yet to be “activated”, so only the first few pixels would see something and the rest would be exposed when the curtain is already closed.
When in electronic shutter though, the pixels must not be activated all at once because then the first row would get less light than the last rows, and this also would mean that the exposure itself is the readout speed which is not the case. So they are activated sequentially, which is in contrast with how it works for mechanical shutter. In this case though, like in my post, how can the image “move” if the pixels are exposed for only 1/1000 s? Is it because the next pixel is exposed for 1/1000 s only after the first one is read, so there is a delay between one pixel being exposed and the next one? Like pixel #2 is not firing for 9/1000 s because it’s waiting for the first one to be read by the 1/10 s readout.
What am I getting wrong here? I’m sure there is some misconception in my mind that is preventing me to see what is going on clearly like everyone else.
I still am not sure how the exposure is started: is it row by row or is it the entire sensor and it’s just the readout that is sequential
The exposure is always started row by row for a progressive (electronic or mechanical) shutter. It’s also read out row by row.
Really truly think of the photo sites as buckets collecting liquid light. It seems like you understand that they all need to be exposed for the same duration to get an even exposure. That doesn’t mean they all need to be read immediately - these buckets don’t leak. In other words, it’s OK for a mechanical shutter to sweep the sensor much faster than the sensor can be read. Those buckets will still have the correct quantity of light in them when they’re (eventually) read.
Step 1: empty the buckets in the dark
Step 2: expose the buckets for the same time duration. Note that this ends with the sensor back in complete darkness
Step 3: measure how full the buckets are
No light hitting the sensor at the end of step 2 means the buckets won’t keep filling while they’re waiting to be read.
I would not understand how it could keep up with the curtains
For a mechanical shutter it doesn’t have to.
So they are activated sequentially, which is in contrast with how it works for mechanical shutter.
For both electronic and mechanical sequential shutters, the rows are still exposed in sequence (the top most row gets its exposure started before the row below it). It is a sequential shutter after all. In the case of a mechanical shutter, this takes 1/250, which is pretty fast but it’s not instant.
In this case though, like in my post, how can the image “move” if the pixels are exposed for only 1/1000 s? Is it because the next pixel is exposed for 1/1000 s only after the first one is read, so there is a delay between one pixel being exposed and the next one? Like pixel #2 is not firing for 9/1000 s because it’s waiting for the first one to be read by the 1/10 s readout
The time delay between exposiyand reading rows is exactly what causes objects to look stretched. The only difference is that sweeping the shutter with the mechanical shutter happens 25 times faster than the electronic one.
Ask more questions, but I think you’re close!
Everytime I feel like it makes perfect sense, I find something that shatters it all.
Here’s what makes sense to me after your explanation: the buckets have the lid off all the time, so anytime light falls on them, it gets collected. This makes perfect sense in mechanical shutter: they are there in complete darkness waiting for light, and when the curtains fly over them some light falls inside the buckets, and this light all fell almost at the same time. Basically they all saw the same thing. After that they stay open, but since it’s total darkness the amount of light stays the same, and the readout can calmly go to each bucket and measure what’s inside. This works perfectly. An assumption I make in this case is that the buckets don’t really have a lid at all, you don’t control when they close, you can only control how much light enters by means of the shutter. Is this assumption correct?
Now, I need to apply the same logic to the electronic shutter because otherwise I go crazy. In the electronic shutter you must have a way to control that lid otherwise you can’t control the exposure. In the mechanical shutter the curtains did the job, in the electronic one light is already falling inside those buckets all the time so you must have a way to limit that, and it cannot be the readout alone because you would be limited to the readout speed as the shortest exposure time, which is not the case.
If you indeed can limit the amount of light, then there are two separate mechanisms to interact with the buckets on the same sensor and in principle this seems weird to me, I feel like this is not correct but I might be mistaken.
With a way to limit the amount of light then it would make sense that, in ES, if I imagine the readout as the thing that adds pieces of the picture sequentially, each bucket waits with its lid closed, it opens the lid for a tiny bit, it’s filled by the set amount of light, and is immediately measured by the readout and a piece is added to the picture, but since it takes a long time to measure this one bucket, the next one just stands waiting with its lid closed. For this reason the moment of the collection of light is limited to the readout speed, and it makes sense that by the time you’re done with all buckets a lot of time has passed because the readout is slow, so the rolling shutter effect happens, but they all had their lid lifted for the set amount of time so the amount of light is correct.
The two scenarios make sense to me only if there are two ways to expose the buckets. And I feel like this is just wrong and I’m missing a tiny piece of information that lets me imagine a single mechanism that can do both.
Sorry for the delayed reply. This deserved a keyboard and monitor. My previous replies have all been typed out on a phone. Brace yourself for a wall of text!
I’m going to reorder your reply a bit here, so bear with my. No words or sentences have been changed, just their order.
Fundamentals first:
Here’s what makes sense to me after your explanation: the buckets have the lid off all the time, so anytime light falls on them, it gets collected. […] An assumption I make in this case is that the buckets don’t really have a lid at all, you don’t control when they close, you can only control how much light enters by means of the shutter. Is this assumption correct?
This is correct. The steps remain:
- Empty the bucket you want to measure
- Allow it to fill for the appropriate amount of time (shutter duration)
- Measure how full the bucket is
A mechanical shutter can provide a lid, but there is not a lid in the case of an electronic shutter.
This makes perfect sense in mechanical shutter: they are there in complete darkness waiting for light, and when the curtains fly over them some light falls inside the buckets, and this light all fell almost at the same time. Basically they all saw the same thing. After that they stay open, but since it’s total darkness the amount of light stays the same, and the readout can calmly go to each bucket and measure what’s inside. This works perfectly.
The key bit of this paragraph is almost. Although 1/250 is pretty darn fast, you can experience rolling shutter with mechanical shutters. Here’s another example. Look at the ball touching the bat in the top of the photo and the shadow of the ball moving away from the bottom. To save you two clicks:
The below was taken on a Canon EOS-1D X, which is a DSLR
Not sure what the photos below were taken on, but it was with the mechanical shutter active
Now, I need to apply the same logic to the electronic shutter because otherwise I go crazy. In the electronic shutter you must have a way to control that lid otherwise you can’t control the exposure. In the mechanical shutter the curtains did the job, in the electronic one light is already falling inside those buckets all the time so you must have a way to limit that, and it cannot be the readout alone because you would be limited to the readout speed as the shortest exposure time, which is not the case.
I think this might be where you’re getting hung up: there is no lid for the buckets when using an electronic shutter. As the electronic shutter sweeps the frame from top to bottom it’s emptying buckets as they’re reached, pausing, allowing them to fill for the exposure duration, measuring how full the bucket is, and then moving on to the next bucket. This is the reason why rolling shutter is so much more pronounced on electronic shutter cameras with slow readout speeds - it has to wait for the prior bucket to be read before emptying the next one and starting the process all over.
To put this in pseudo code for a mechanical shutter:
empty all buckets // fast sweep mechanical shutter across lens // 1/250 second in total for each bucket in buckets: read the value // no light hits any bucket during this processA nd an electronic shutter
for each bucket in buckets: empty bucket // fast allow bucket to fill for exposure duration // usually pretty fast, but note that you're going to be doing this a lot read how full the bucket is // slow and you're also going to be doing this a lotWith a way to limit the amount of light then it would make sense that, in ES, if I imagine the readout as the thing that adds pieces of the picture sequentially, each bucket waits with its lid closed, it opens the lid for a tiny bit,
With an ES there is no way to close the bucket, but you can empty them on demand
it’s filled by the set amount of light, and is immediately measured by the readout and a piece is added to the picture, but since it takes a long time to measure this one bucket,
This is accurate! You’re very close
the next one just stands waiting with its lid closed.
The next one is just hanging out filling (or maybe even saturating), but it doesn’t matter because it’s going to get emptied before it gets measured.
For this reason the moment of the collection of light is limited to the readout speed, and it makes sense that by the time you’re done with all buckets a lot of time has passed because the readout is slow, so the rolling shutter effect happens
This is also accurate! But also: rolling shutter can totally happen with mechanical shutters too. The only way to truly eliminate it is by using a global shutter. If you want to see one in action, the only video I can vouch for is Chris and Jordan’s first impressions video on the A9III. It’s very weird seeing a lot of high speed sports with zero distortion.
but they all had their lid lifted for the set amount of time so the amount of light is correct
I hope you’ve got it by this point, but the way we control the buckets is by dumping on demand and then measuring right after their exposure period has elapsed.
First of all, your replies are worth waiting for :) It’s not the first time we interact, maybe you didn’t notice but I sure did and I’m glad.
I should have specified, I understand that in MS rolling shutter is just hard to get, but not impossible. After all, the curtains don’t sweep the sensor instantaneously. But I thank you very much for the detailed demonstration!
In general, I think I’ve figured it out now. The buckets are always open, but I had it backwards: we don’t have lids to close to limit the exposure, we straight up flip the buckets and empty them to control how much light they contain. (Btw, I love the pseudo code you put there. My mind works a lot “by rules” and that code example is spot on, it’s crystal clear to me. Thank you).
Let me try to explain so you can tell me if I understood or not:
-
In MS, the buckets are always open, in total darkness, until the shutter flies above them and lets a limited amount of light fall into them. The shutter is so fast, that the image each bucket sees is very chronologically close to the other ones. Once the shutter has passed, the buckets are back to total darkness, so no other light is collected, and the readout can piece together the light collected by every bucket. It does it slowly, but since no light is coming in anymore this does not cause any “distortion”. Important point: the readout does not empty each bucket before reading it. It arrives at the next bucket it and measures the content.
-
In ES, the buckets are always open, in total light, then the readout starts and at each bucket, it empties it (it’s probably full but since it’s going to be emptied nobody cares nor sees what was collected) and then flips it back up for the specified amount of time before immediately reading it, then it goes on to the next bucket. This way, the light read by the readout is always the correct amount, but since it’s collected at the very slow readout speed, the image that is pieced together will be chronologically skewed in the direction of the readout (top of the image is “before” the bottom in a tangible way). Important point: the readout does empty each bucket before reading it. It arrives at the next bucket, it empties it, and waits for the specified time interval before measuring the content.
So I guess the last piece I need is: there is a “way” to choose whether to empty the charge accumulated by each pixel when reading it or not. In MS it’s chosen (at some mechanical level I suppose) that the readoud doesn’t empty the charge accumulated before measuring it at each pixel, in ES it does empty it before measuring it. Not only that, in MS it’s set that the readout happens immediately, while in ES it happens only after having waited the exposure time.
I’m way out of my element when it comes to microelectronics, circuitry, and engineering in this field (I don’t even know what it’s called), so probably even if I got a detailed answer to this I wouldn’t be able to fully comprehend it.
If my intuition is correct, then I can say I finally have an answer to this (for me) burning question I’ve had for a while. I guess at the end there is indeed a fundamental difference in how the sensor works between MS and ES. I feel good thinking that at least my hunch was right, what I imagined was just so much more complicated that it felt impossible to be correct. Your explanation is much more “acceptable” for my brain :)
Apologies again for the delay, but I’m flattered to hear that you find the replies useful.
I should have specified, I understand that in MS rolling shutter is just hard to get, but not impossible.
The reason why I keep harping on this is because ES vs MS are really very similar in how they operate. Empty, expose, measure. The only thing that’s really different between the two is a MS can cover the photo sites after expose, which enables the shutter curtains to be ahead of the sensor’s readout position.
In MS, the buckets are always open, in total darkness, until the shutter flies above them and lets a limited amount of light fall into them.
Gotta empty the buckets ahead of time, but yup.
In MS, the buckets are always open, in total darkness, until the shutter flies above them and lets a limited amount of light fall into them. (…) Once the shutter has passed, the buckets are back to total darkness, so no other light is collected, and the readout can piece together the light collected by every bucket. It does it slowly, but since no light is coming in anymore this does not cause any “distortion”.
Yup!
The ES section is accurate as well, but…
(in your MS section) The shutter is so fast, that the image each bucket sees is very chronologically close to the other ones.
and
(in your ES section) This way, the light read by the readout is always the correct amount, but since it’s collected at the very slow readout speed, the image that is pieced together will be chronologically skewed in the direction of the readout (top of the image is “before” the bottom in a tangible way).
Really truly: some ES shutters are just as fast, or nearly as fast, as a mechanical shutter. That’s why some cameras are shipping without mechanical shutters now. Off the top of my head, some of the fastest reading sensors are: Z8/Z9 (1/250 and no shutter), A1 (1/240 and no mechanical front curtain), A9 i/ii (1/160), X-H2s (1/150), OM-1 (1/120), R3 (1/105).
So I guess the last piece I need is: there is a “way” to choose whether to empty the charge accumulated by each pixel when reading it or not.
I’m not sure if reading is used to empty the photo site, or if there’s another mechanism to be honest. Either way, you need to empty, fill for the exposure duration, and read. If the second read happens to empty the site again it doesn’t really matter that much.
In MS it’s chosen (at some mechanical level I suppose) that the readoud doesn’t empty the charge accumulated before measuring it at each pixel, in ES it does empty it before measuring it. Not only that, in MS it’s set that the readout happens immediately, while in ES it happens only after having waited the exposure time.
The sensor is completely electrical, the big difference between ES and MS is the shutter mechanism. ES = use the sensor as the shutter. MS = use two mechanical curtains. In mechanical mode the photo sites can still be thought of as buckets. You gotta empty them before you expose them for the proper duration, otherwise you’ll wind up with an improperly exposed image.
I’m way out of my element when it comes to microelectronics, circuitry, and engineering in this field (I don’t even know what it’s called), so probably even if I got a detailed answer to this I wouldn’t be able to fully comprehend it.
We’re not talking anything beyond logical reasoning here. If you’ve been exposed to that mindset before, or just happen to think that way, you can probably wrap your head around this.
I guess at the end there is indeed a fundamental difference in how the sensor works between MS and ES.
The big difference is that mechanical shutters can put a lid on the buckets, which enables the shutter to be ahead of the readout. The basic pattern is still the same: empty, expose, measure.
Just wait untli you start thinking about Electronic Front-Curtain mechanical shutters. In this case, the camera only uses the rear mechanical shutter curtain to block the senor - the front curtain uses the ES methodology.
Remember: empty, expose, fill.
Heh, sorry if I sound so pedantic, but I thought that would be my final message. Let me try with this one.
Yes, the steps are empty, expose, measure. I was just trying to explain the difference I’m finding in the electronic handling of these steps between MS and ES. In both cases though, the pixels experience the same three steps in the same sequence: empty, expose, measure.
The “only” difference I’m finding is that in MS the action of “measuring” the pixel happens effectively differently than in ES because, at each photo site, the action of “measuring” is only, well, reading sequentially the amount collected, which is limited in speed to the readout speed.
In ES, the action of “measuring” is preceded every time by "emptying, exposing"sequentially, all controlled electronically.
This makes me think that there is some liberty in choosing when to empty and expose pixels electronically, and it’s not really limited neither by some kind of speed, nor some kind of sequence: in MS you don’t really care because the curtains do the job, in ES you must control precisely when to empty a pixel and when to expose it.
This would also agree with how I understand EFC-S works: your closing of the curtains is limited by the curtain speed, so you have to empty and expose those pixels at the right time to let the curtain end the exposure so that you collect the specified amount of light. So if you’re shooting at 1/1000 and the curtain closes at 1/250, you follow the ES method and empty+expose those buckets a tiny bit before the curtain passes over them. And since after the curtain it’s darkness, just like in MS you can wait for the readout without worry.
Therefore, it is to my understanding that, electronically speaking, emptying and exposing the pixels can happen at very high speeds, independently from the measuring step: in MS, you empty and expose all of the pixels at once (from the sensor’s POV, of course it’s the curtain that does the exposure job), and then you measure them - the measuring is done once, “alone”; in ES you empty, expose, and measure each pixel individually - the measuring is coordinated with emptying and exposing (which also agrees with your beautiful pseudo-code).
EDIT: maybe it can be reduced to the emptying action alone. You can empty whenever you want, and since the pixel always gathers light as soon as you’re done emptying it, as long as you time your emptying action correctly the exposure can happen however you prefer: by the curtains in MS, by emptying at the right time so that the pixel is exposed correctly for when the readout happens in ES, and right before the curtain passes over in EFC-S
To me, in EFC-S this takes the best of both worlds: no fear of distortion because the reading phase is done in darkness, higher theoretical shutter (i.e. exposure) speed because you can empty+expose each pixel electronically even almost as the curtain closes above them.
I hope you understand that i truly appreciate your help, and I’m sorry I keep hammering at this stuff. I truly appreciate all the patience you can have for me.
-
The photosites are continuously reading the light falling on them. When you take a picture, the system takes discreet readings of those values for the length of your shutter speed. However it can only handle the information from a portion of the sensor at a time. So it reads in sequential stripes. The longer the exposure time the more of a chance that the sensor or the subject has moved before the camera reads all the stripes.
Thanks for the reply. May I ask you to go into more details?
You know how a strobe light can make things look like its sort of jumping through time? Sort of make it look like there’s a way lower frame rate irl? It’s like that but imagine instead of the light flashing on and off, it rolls from top to bottom.
