Skip to content

Commit 130029a

Browse files
naushirnaushir
committed
Merge branch 'master' into next
Signed-off-by: Naushir Patuck <[email protected]>
2 parents 2915667 + 5e68727 commit 130029a

File tree

120 files changed

+2909
-806
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

120 files changed

+2909
-806
lines changed

‎Documentation/design/ae.rst‎

Lines changed: 331 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,331 @@
1+
.. SPDX-License-Identifier: CC-BY-SA-4.0
2+
3+
Design of Exposure and Gain controls
4+
====================================
5+
6+
This document explains the design and rationale of the controls related to
7+
exposure and gain. This includes the all-encompassing auto-exposure (AE), the
8+
manual exposure control, and the manual gain control.
9+
10+
Description of the problem
11+
--------------------------
12+
13+
Sub controls
14+
^^^^^^^^^^^^
15+
16+
There are more than one control that make up total exposure: exposure time,
17+
gain, and aperture (though for now we will not consider aperture). We already
18+
had individual controls for setting the values of manual exposure and manual
19+
gain, but for switching between auto mode and manual mode we only had a
20+
high-level boolean AeEnable control that would set *both* exposure and gain to
21+
auto mode or manual mode; we had no way to set one to auto and the other to
22+
manual.
23+
24+
So, we need to introduce two new controls to act as "levers" to indicate
25+
individually for exposure and gain if the value would come from AEGC or if it
26+
would come from the manual control value.
27+
28+
Aperture priority
29+
^^^^^^^^^^^^^^^^^
30+
31+
We eventually may need to support aperture, and so whatever our solution is for
32+
having only some controls on auto and the others on manual needs to be
33+
extensible.
34+
35+
Flickering when going from auto to manual
36+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
37+
38+
When a manual exposure or gain value is requested by the application, it costs
39+
a few frames worth of time for them to take effect. This means that during a
40+
transition from auto to manual, there would be flickering in the control values
41+
and the transition won't be smooth.
42+
43+
Take for instance the following flow, where we start on auto exposure (which
44+
for the purposes of the example increments by 1 each frame) and we want to
45+
switch seamlessly to manual exposure, which involves copying the exposure value
46+
computed by the auto exposure algorithm:
47+
48+
::
49+
50+
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
51+
| N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |
52+
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
53+
54+
Mode requested: Auto Auto Auto Manual Manual Manual Manual
55+
Exp requested: N/A N/A N/A 2 2 2 2
56+
Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8
57+
58+
Mode used: Auto Auto Auto Auto Auto Manual Manual
59+
Exp used: 0 1 2 3 4 2 2
60+
61+
As we can see, after frame N+2 completes, we copy the exposure value that was
62+
used for frame N+2 (which was computed by AE algorithm), and queue that value
63+
into request N+3 with manual mode on. However, as it takes two frames for the
64+
exposure to be set, the exposure still changes since it is set by AE, and we
65+
get a flicker in the exposure during the switch from auto to manual.
66+
67+
A solution is to *not submit* any exposure value when manual mode is enabled,
68+
and wait until the manual mode as been "applied" before copying the exposure
69+
value:
70+
71+
::
72+
73+
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
74+
| N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |
75+
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
76+
77+
Mode requested: Auto Auto Auto Manual Manual Manual Manual
78+
Exp requested: N/A N/A N/A None None None 5
79+
Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8
80+
81+
Mode used: Auto Auto Auto Auto Auto Manual Manual
82+
Exp used: 0 1 2 3 4 5 5
83+
84+
In practice, this works. However, libcamera has a policy where once a control
85+
is submitted, its value is saved and does not need to be resubmitted. If the
86+
manual exposure value was set while auto mode was on, in theory the value would
87+
be saved, so when manual mode is enabled, the exposure value that was
88+
previously set would immediately be used. Clearly this solution isn't correct,
89+
but it can serve as the basis for a proper solution, with some more rigorous
90+
rules.
91+
92+
Existing solutions
93+
------------------
94+
95+
Raspberry Pi
96+
^^^^^^^^^^^^
97+
98+
The Raspberry Pi IPA gets around the lack of individual AeEnable controls for
99+
exposure and gain by using magic values. When AeEnable is false, if one of the
100+
manual control values was set to 0 then the value computed by AEGC would be
101+
used for just that control. This solution isn't desirable, as it prevents
102+
that magic value from being used as a valid value.
103+
104+
To get around the flickering issue, when AeEnable is false, the Raspberry Pi
105+
AEGC simply stops updating the values to be set, without restoring the
106+
previously set manual exposure time and gain. This works, but is not a proper
107+
solution.
108+
109+
Android
110+
^^^^^^^
111+
112+
The Android HAL specification requires that exposure and gain (sensitivity)
113+
must both be manual or both be auto. It cannot be that one is manual while the
114+
other is auto, so they simply don't support sub controls.
115+
116+
For the flickering issue, the Android HAL has an AeLock control. To transition
117+
from auto to manual, the application would keep AE on auto, and turn on the
118+
lock. Once the lock has propagated through, then the value can be copied from
119+
the result into the request and the lock disabled and the mode set to manual.
120+
121+
The problem with this solution is, besides the extra complexity, that it is
122+
ambiguous what happens if there is a state transition from manual to locked
123+
(even though it's a state transition that doesn't make sense). If locked is
124+
defined to "use the last automatically computed values" then it could use the
125+
values from the last time it AE was set to auto, or it would be undefined if AE
126+
was never auto (eg. it started out as manual), or if AE is implemented to run
127+
in the background it could just use the current values that are computed. If
128+
locked is defined to "use the last value that was set" there would be less
129+
ambiguity. Still, it's better if we can make it impossible to execute this
130+
nonsensical state transition, and if we can reduce the complexity of having
131+
this extra control or extra setting on a lever.
132+
133+
Summary of goals
134+
----------------
135+
136+
- We need a lock of some sort, to instruct the AEGC to not update output
137+
results
138+
139+
- We need manual modes, to override the values computed by the AEGC
140+
141+
- We need to support seamless transitions from auto to manual, and do so
142+
without flickering
143+
144+
- We need custom minimum values for the manual controls; that is, no magic
145+
values for enabling/disabling auto
146+
147+
- All of these need to be done with AE sub-controls (exposure time, analogue
148+
gain) and be extensible to aperture in the future
149+
150+
Our solution
151+
------------
152+
153+
A diagram of our solution:
154+
155+
::
156+
157+
+----------------------------+-------------+------------------+-----------------+
158+
| INPUT | ALGORITHM | RESULT | OUTPUT |
159+
+----------------------------+-------------+------------------+-----------------+
160+
161+
ExposureTimeMode ExposureTimeMode
162+
---------------------+----------------------------------------+----------------->
163+
0: Auto | |
164+
1: Manual | V
165+
| |\
166+
| | \
167+
| /----------------------------------> | 1| ExposureTime
168+
| | +-------------+ exposure time | | -------------->
169+
\--)--> | | --------------> | 0|
170+
ExposureTime | | | | /
171+
------------------------+--> | | |/
172+
| | AeState
173+
| AEGC | ----------------------------------->
174+
AnalogueGain | |
175+
------------------------+--> | | |\
176+
| | | | \
177+
/--)--> | | --------------> | 0| AnalogueGain
178+
| | +-------------+ analogue gain | | -------------->
179+
| \----------------------------------> | 1|
180+
| | /
181+
| |/
182+
| ^
183+
AnalogueGainMode | | AnalogueGainMode
184+
---------------------+----------------------------------------+----------------->
185+
0: Auto
186+
1: Manual
187+
188+
AeEnable
189+
- True -> ExposureTimeMode:Auto + AnalogueGainMode:Auto
190+
- False -> ExposureTimeMode:Manual + AnalogueGainMode:Manual
191+
192+
193+
The diagram is divided in four sections horizontally:
194+
195+
- Input: The values received from the request controls
196+
197+
- Algorithm: The algorithm itself
198+
199+
- Result: The values calculated by the algorithm
200+
201+
- Output: The values reported in result metadata and applied to the device
202+
203+
The four input controls are divided between manual values (ExposureTime and
204+
AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The
205+
former are the manual values, the latter control how they're applied. The two
206+
modes are independent from each other, and each can take one of two values:
207+
208+
- Auto (0): The AGC computes the value normally. The AGC result is applied
209+
to the output. The manual value is ignored *and is not retained*.
210+
211+
- Manual (1): The AGC uses the manual value internally. The corresponding
212+
manual control from the request is applied to the output. The AGC result
213+
is ignored.
214+
215+
The AeState control reports the state of the unified AEGC block. If both
216+
ExposureTimeMode and AnalogueGainMode are set to manual then it will report
217+
Idle. If at least one of the two is set to auto, then AeState will report
218+
if the AEGC has Converged or not (Searching). This control replaces the old
219+
AeLocked control, as it was insufficient for reporting the AE state.
220+
221+
There is a caveat to manual mode: the manual control value is not retained if
222+
it is set during auto mode. This means that if manual mode is entered without
223+
also setting the manual value, then it will enter a state similar to "locked",
224+
where the last automatically computed value while the mode was auto will be
225+
used. Once the manual value is set, then that will be used and retained as
226+
usual.
227+
228+
This simulates an auto -> locked -> manual or auto -> manual state transition,
229+
and makes it impossible to do the nonsensical manual -> locked state
230+
transition.
231+
232+
AeEnable still exists to allow applications to set the mode of all the
233+
sub-controls at once. Besides being for convenience, this will also be useful
234+
when we eventually implement an aperture control. This is because applications
235+
that will be made before aperture will have been available would still be able
236+
to set aperture mode to auto or manual, as opposed to having the aperture stuck
237+
at auto while the application really wanted manual. Although the aperture would
238+
still be stuck at an uncontrollable value, at least it would be at a static
239+
usable value as opposed to varying via the AEGC algorithm.
240+
241+
With this solution, the earlier example would become:
242+
243+
::
244+
245+
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
246+
| N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|
247+
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
248+
Mode requested: Auto Manual Manual Manual Manual Manual Manual Manual Manual
249+
Exp requested: N/A None None None None 10 None 10 10
250+
Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+11 N+12
251+
252+
Mode used: Auto Auto Auto Manual Manual Manual Manual Manual Manual
253+
Exp used: 2 3 4 5 5 5 5 10 10
254+
255+
This example is extended by a few frames to exhibit the simulated "locked"
256+
state. At frame N+5 the application has confirmed that the manual mode has been
257+
entered, but does not provide a manual value until request N+7. Thus, the value
258+
that is used in requests N+5 and N+6 (where the mode is disabled), comes from
259+
the last value that was used when the mode was auto, which comes from frame
260+
N+4.
261+
262+
Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for
263+
the exposure to be applied. N+8 does not supply a manual value, but the last
264+
supplied value is retained, so a manual value of 10 is still used and set in
265+
frame N+10.
266+
267+
Although this behavior is the same as what we had with waiting for the manual
268+
mode to propagate (in the section "Description of the problem"), this time it
269+
is correct as we have defined specifically that if a manual value was specified
270+
while the mode was auto, it will not be retained.
271+
272+
Description of the controls
273+
---------------------------
274+
275+
As described above, libcamera offers the following controls related to exposure
276+
and gain:
277+
278+
- AnalogueGain
279+
280+
- AnalogueGainMode
281+
282+
- ExposureTime
283+
284+
- ExposureTimeMode
285+
286+
- AeState
287+
288+
- AeEnable
289+
290+
Auto-exposure and auto-gain can be enabled and disabled separately using the
291+
ExposureTimeMode and AnalogueGainMode controls respectively. The AeEnable
292+
control can also be used, as it sets both of the modes simultaneously. The
293+
AeEnable control is not returned in metadata.
294+
295+
When the respective mode is set to auto, the respective value that is computed
296+
by the AEGC algorithm is applied to the image sensor. Any value that is
297+
supplied in the manual ExposureTime/AnalogueGain control is ignored and not
298+
retained. Another way to understand this is that when the mode transitions from
299+
auto to manual, the internally stored control value is overwritten with the
300+
last value computed by the auto algorithm.
301+
302+
This means that when we transition from auto to manual without supplying a
303+
manual control value, the last value that was set by the AEGC algorithm will
304+
keep be used. This can be used to do a flickerless transition from auto to
305+
manual as described earlier. If the camera started out in manual mode and no
306+
corresponding value has been supplied yet, then a best-effort default value
307+
shall be set.
308+
309+
The manual control value can be set in the same request as setting the mode to
310+
auto if the desired manual control value is already known.
311+
312+
Transitioning from manual to auto shall be implicitly flickerless, as the AEGC
313+
algorithms are expected to start running from the last manual value.
314+
315+
The AeState metadata reports the state of the AE algorithm. As AE cannot
316+
compute exposure and gain separately, the state of the AE component is
317+
unified. There are three states: Idle, Searching, and Converged.
318+
319+
The state shall be Idle if both ExposureTimeMode and AnalogueGainMode
320+
are set to Manual. If the camera only supports one of the two controls,
321+
then the state shall be Idle if that one control is set to Manual. If
322+
the camera does not support Manual for at least one of the two controls,
323+
then the state will never be Idle, as AE will always be running.
324+
325+
The state shall be Searching if at least one of exposure or gain calculated
326+
by the AE algorithm is used (that is, at least one of the two modes is Auto),
327+
*and* the value(s) have not converged yet.
328+
329+
The state shall be Converged if at least one of exposure or gain calculated
330+
by the AE algorithm is used (that is, at least one of the two modes is Auto),
331+
*and* the value(s) have converged.

‎Documentation/environment_variables.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,8 +57,8 @@ LIBCAMERA_RPI_CONFIG_FILE
5757

5858
Example value: ``/usr/local/share/libcamera/pipeline/rpi/vc4/minimal_mem.yaml``
5959

60-
LIBCAMERA_RPI_TUNING_FILE
61-
Define a custom JSON tuning file to use in the Raspberry Pi.
60+
LIBCAMERA_<NAME>_TUNING_FILE
61+
Define a custom IPA tuning file to use with the pipeline handler `NAME`.
6262

6363
Example value: ``/usr/local/share/libcamera/ipa/rpi/vc4/custom_sensor.json``
6464

‎Documentation/guides/application-developer.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -128,7 +128,7 @@ available.
128128
129129
std::string cameraId = cameras[0]->id();
130130
131-
auto camera = cm->get(cameraId);
131+
camera = cm->get(cameraId);
132132
/*
133133
* Note that `camera` may not compare equal to `cameras[0]`.
134134
* In fact, it might simply be a `nullptr`, as the particular

‎Documentation/index.rst‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,9 @@
2323
SoftwareISP Benchmarking <software-isp-benchmarking>
2424
Tracing guide <guides/tracing>
2525

26+
Design document: AE <design/ae>
27+
2628
.. toctree::
2729
:hidden:
2830

29-
introduction
31+
introduction

‎Documentation/meson.build‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,7 @@ if sphinx.found()
128128
'coding-style.rst',
129129
'conf.py',
130130
'contributing.rst',
131+
'design/ae.rst',
131132
'documentation-contents.rst',
132133
'environment_variables.rst',
133134
'feature_requirements.rst',

‎include/libcamera/base/compiler.h‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

0 commit comments

Comments
 (0)