The API is very similar for encoding/decoding and audio/video, and works as follows:
At the beginning of decoding or encoding, the codec might accept multiple input frames/packets without returning a frame, until its internal buffers are filled. This situation is handled transparently if you follow the steps outlined above.
In theory, sending input can result in EAGAIN - this should happen only if not all output was received. You can use this to structure alternative decode or encode loops other than the one suggested above. For example, you could try sending new input on each iteration, and try to receive output if that returns EAGAIN.
End of stream situations. These require "flushing" (aka draining) the codec, as the codec might buffer multiple frames or packets internally for performance or out of necessity (consider B-frames). This is handled as follows:
Using the API as outlined above is highly recommended. But it is also possible to call functions outside of this rigid schema. For example, you can call avcodec_send_packet() repeatedly without calling avcodec_receive_frame(). In this case, avcodec_send_packet() will succeed until the codec's internal buffer has been filled up (which is typically of size 1 per output frame, after initial input), and then reject input with AVERROR(EAGAIN). Once it starts rejecting input, you have no choice but to read at least some output.
Not all codecs will follow a rigid and predictable dataflow; the only guarantee is that an AVERROR(EAGAIN) return value on a send/receive call on one end implies that a receive/send call on the other end will succeed. In general, no codec will permit unlimited buffering of input or output.
This API replaces the following legacy functions:
Mixing new and old function calls on the same AVCodecContext is not allowed, and will result in arbitrary behavior.
Some codecs might require using the new API; using the old API will return an error when calling it. All codecs support the new API.
A codec is not allowed to return EAGAIN for both sending and receiving. This would be an invalid state, which could put the codec user into an endless loop. The API has no concept of time either: it cannot happen that trying to do avcodec_send_packet() results in EAGAIN, but a repeated call 1 second later accepts the packet (with no other receive/flush API calls involved). The API is a strict state machine, and the passage of time is not supposed to influence it. Some timing-dependent behavior might still be deemed acceptable in certain cases. But it must never result in both send/receive returning EAGAIN at the same time at any point. It must also absolutely be avoided that the current state is "unstable" and can "flip-flop" between the send/receive APIs allowing progress. For example, it's not allowed that the codec randomly decides that it actually wants to consume a packet now instead of returning a frame, after it just returned EAGAIN on an avcodec_send_packet() call.