Buckets and Bucket Brigades

Bucketsprovide uniform access to a variety of data sources. Bucket
brigadesare collections of bucketsusually use to represent a block of data.
For instance, a bucket brigade may contain buckets of headers allocated from a request pool followed by a bucket representing a local file. A filter can add data before or after this file block without having to copy the the data. apr_brigade_to_iovec makes it easy to pass this scatter-write paradigm through the sockets layer all the way to the ethernet and disk controllers.

Table of contents

• Types of buckets
• Bucket virual functions
• Using buckets and bucket brigades
• Bucket type and function names
• Calling conventions and contracts
• Common idioms
• Brigade functions
• Brigade creation and destruction
• Adding data to brigades
• Brigade data consumption
• Misc
• Brigade macros
• Bucket list manipluation macros
• ...

Types of buckets

Apache provides a set of standard bucket types allowing data collection from a variety of sources:

transient  (const char *buf, apr_size_t nbyte)

Represents a data allocated off the stack. When the setaside function is called, this data is copied on to the heap.

暂时 (const char *buf,apr_size_t nbyte)

表示从堆栈上分配的数据。当setaside函数调用时，数据被拷贝到堆里。

heap  (const char *buf, apr_size_t nbyte, int copy, apr_size_t *w)

Represents a data allocated from the heap.

堆 (const char *buf,apr_size_t nbyte,int copy,apr_size_t*w)

表示从堆内分配的数据。

pool  (const char *buf, apr_size_t length, apr_pool_t *pool)

Represents a data that was allocated from a pool. IF this bucket is still available when the pool is cleared, the data is copied on to the heap.

池 (const char *buf,apr_size_t length,apr_pool_t*pool)

表示从一个内存池上分配的数据。如果内存池在清除时桶依然有效，则数据被拷贝到堆内。

file  (apr_file_t *fd, apr_off_t offset, apr_size_t len)

Represents a file on disk.

文件 (apr_file_t*fd,apr_off_t offset,apr_size_t len)

表示磁盘上文件。

mmap  (apr_mmap_t *mm, apr_off_t start, apr_size_t length)

This bucket represents an MMAP'ed file.

mmap (apr_mmap_t*mm,apr_off_t start,apr_size_t length)

mmap允许直接将设备内存映射到用户进程的地址空间中。

表示MMAP’en文件

socket   (apr_socket_t *thissock)

Represents a socket connection to another machine.

socket (apr_socket_t*thissock)

表示连接到其它机器的一个socket

pipe   (apr_file_t *thispipe)

Represents a pipe to another program.

pipe (apr_file_t*thispipe)

表示一个连接到其它程序的一个pipe。

Marker buckets

eos  (void)

Signifies that there will be no more data, ever. All filters MUST send all data to the next filter when they receive a bucket of this type.

flush  (void)

Signifies that all data should be flushed to the next filter. The flush bucket should be sent with the other buckets.

标志桶

eos (void)

表示没有后续数据。所有的过滤器在收到这种类型的桶时，必须传送所有的数据到下一个过滤器。

flush (void)

表示所有数据需要刷新到下一个过滤器。刷新桶必须和其它类型的桶同时使用。

Opaque buckets

immortal  (const char *buf, apr_size_t nbyte)

Represents a segment of data that the creator is willing to take responsability for. The core will do nothing with the data in an immortal bucket.

永久桶

immortal (const char*buf,apr_size_t nbyte)

表示一个数据节，此节的创建者将对其负责。核心模块将对不透明桶不做处理

Bucket virual functions

Buckets implement a set of virtual functions so access and state in different bucket types may be handled uniformly. Following is a list of parameters and a description of each type of bucket followed by the funtion to be used if the bucket does not implement the virtual function:

桶的虚函数

桶实现了一系列的虚函数，这使对不同类型桶的存取和状态统计能统一处理。下面是一些函数说明：

apr_bucket_read  (apr_bucket *b, const char **str, apr_size_t *len, apr_read_type_e block)

Returns the address and size of the data in the bucket. If the data isn't in memory then it is read in and the bucket changes type so that it can refer to the new location of the data. If all the data doesn't fit in the bucket then a new bucket is inserted into the brigade to hold the rest of it.

must be implemented

apr_bucket_split  (apr_bucket *e, apr_off_t point)

Divides the data in a bucket into two regions. After a split the original bucket refers to the first part of the data and a new bucket inserted into the brigade after the original bucket refers to the second part of the data. Reference counts are maintained as necessary. If this is not implemented, apr_brigade_partition simply reads the bucket and creates new ones with the contents from either side of the divider.

apr_bucket_split_notimpl

apr_bucket_setaside  (apr_bucket *e)

Ensures that the data in the bucket has a long enough lifetime. Sometimes it is convenient to create a bucket referring to data on the stack in the expectation that it will be consumed (output to the network) before the stack is unwound. If that expectation turns out not to be valid, the setaside function is called to move the data somewhere safer.

apr_bucket_setaside_notimpl

apr_bucket_copy  (apr_bucket *e, apr_bucket **c)

Makes a duplicate of the bucket structure as long as it's possible to have multiple references to a single copy of the data itself. Not all bucket types can be copied.

apr_bucket_copy_notimpl

apr_bucket_destroy  (void *data)

Maintains the reference counts on the resources used by a bucket and frees them if necessary.

apr_bucket_destroy_notimpl

apr_bucket_delete  (void *data)

@<protected>@<protected>@<protected> calls apr_bucket_destroy.

implemented via a macro

Note: all of the above except destroyand deletereturn a apr_status_t.

Because some buckets have unimplemented functions, and some do not maintain a length, callers must be prepared to work around these limitations.

Using buckets and bucket brigades

This section will (attempt to) provide an outline of the uses of buckets and a small vocabulary of idioms used to accomplish common tasks.

Bucket type and function names

There are strict naming conventions for the bucket types. The typedef name can be found by prepending the name with "apr_bucket_type_", for example, the pool bucket:

b.type = apr_bucket_type_pool

The create function, takes the prototype described above, eg:

apr_bucket * apr_bucket_pool_create(const char *buf, apr_size_t nbyte, apr_pool_t *pool);

The make function, recommended only for bucket implementers, takes a pointer to an apr_bucketplus the prototype described above:

apr_bucket * apr_bucket_pool_make(apr_bucket *b, const char *buf, apr_size_t nbyte, apr_pool_t *pool);

Each type also has a macro to test type. For brevity and maintenance purposes, it is recommended that you use this macro and not manipulation of the bucket data structure.

APR_BUCKET_IS_POOL(bucket)

Calling conventions and contracts

filter conventions dictate the use of buckets. Filters that manipulate a brigade may rely on brigade and bucket functions but are expected to deliver a coherent bucket to subsequent filters via ap_pass_brigade. Filters may be called repeatedly but must be passed a brigade ending with an EOS on the last call.

Common idioms

default_handlercreates a new bucket brigade and adds a file bucket - from server/core.c

bb = apr_brigade_create(r->pool);
e = apr_bucket_file_create(fd, 0, r->finfo.size);      
APR_BRIGADE_INSERT_HEAD(bb, e);
e = apr_bucket_eos_create();
APR_BRIGADE_INSERT_TAIL(bb, e);

ap_http_header_filterlooks through all the buckets for a particular kind: - from modules/http/http_protocol.c

apr_status_t ap_http_header_filter(ap_filter_t *f, apr_bucket_brigade *b)
{
...
apr_bucket *e;
...
APR_BRIGADE_FOREACH(e, b) {
if (e->type == &ap_bucket_type_error) {
ap_bucket_error *eb = e->data;
ap_die(eb->status, f->r);
return AP_FILTER_ERROR;
}
}

ap_content_length_filtermaintains context between calls - from server/protocol.c

ctx = f->ctx;
if (!ctx) { /* first time through */
f->ctx = ctx = apr_pcalloc(r->pool, sizeof(struct content_length_ctx));
}
...
APR_BRIGADE_FOREACH(e, b) 
{
apr_size_t length;
...
length = e->length;
...
ctx->curr_len += length;
r->bytes_sent += length;
}

ap_content_length_filterhandles buckets that don't know their length - from server/protocol.c

if (e->length == -1) { /* if length unknown */
rv = apr_bucket_read(e, &ignored, &length, APR_BLOCK_READ);
if (rv != APR_SUCCESS) {
return rv;
}
}
else {
length = e->length;
}

Brigade functions

Following is a list of the bucket brigade API functions:

Brigade creation and destruction

apr_bucket_brigade * apr_brigade_create (apr_pool_t *p)

Create a new bucket brigade. The resulting brigade's cleanup is registered with p.

apr_status_t apr_brigade_destroy (apr_bucket_brigade *b)

Destroy entire bucket brigade b. This includes destroying all of the buckets within the bucket brigade's bucket list.

apr_status_t apr_brigade_cleanup (void *b)

Empty out an entire bucket brigade. This includes destroying all of the buckets within the bucket brigade's bucket list. This is similar to apr_brigade_destroy(), except that it does not deregister the brigade's pool cleanup function.

Adding data to brigades

int apr_brigade_vputstrs (apr_bucket_brigade *b, apr_brigade_flush flush, void *ctx, va_list va)

Write strings of data indicated by va into brigade b. If the flush is non-NULL and b is empty and the length of a string in va is greater than APR_BUCKET_BUFF_SIZE, flush is called with context ctx.

int apr_brigade_write (apr_bucket_brigade *b, apr_brigade_flush flush, void *ctx, const char *str, apr_size_t nbyte)

Writes nbyte bytes from str into brigade b. If the flush is non-NULL and b is empty and nbyte is greater than APR_BUCKET_BUFF_SIZE, flush is called with context ctx.

int apr_brigade_puts (apr_bucket_brigade *b, apr_brigade_flush flush, void *ctx, const char *str)

Write zero-terminated string str into brigade b. If the flush is non-NULL and b is empty and strlen(str) is greater than APR_BUCKET_BUFF_SIZE, flush is called with context ctx.

int apr_brigade_putc (apr_bucket_brigade *b, apr_brigade_flush flush, void *ctx, const char c)

Write character c into brigade b. flush and ctx are just window dressing.

int apr_brigade_vprintf (apr_bucket_brigade *b, apr_brigade_flush flush, void *ctx, const char *fmt, va_list va)

Writes string resulting from vprintf(fmt, va) into brigade b. If the flush is non-NULL and b is empty and the length of the resulting string is greater than APR_BUCKET_BUFF_SIZE, flush is called with context ctx.

Brigade data consumption

apr_bucket_brigade * apr_brigade_split (apr_bucket_brigade *b, apr_bucket *e)

Split a bucket brigade into two, such that the given bucket is the first in the new bucket brigade. This function is useful when a filter wants to pass only the initial part of a brigade to the next filter. Note, individual buckets may also be split.

apr_bucket * apr_brigade_partition (apr_bucket_brigade *b, apr_off_t point)

Partition a bucket brigade at a given offset (in bytes from the start of the brigade). This is useful whenever a filter wants to use known ranges of bytes from the brigade; the ranges can even overlap.

void apr_brigade_consumeapr_bucket_brigade *b, int nbytes)

Consume nbytes from beginning of b -- call apr_bucket_destroy as appropriate, and/or modify start on last element. Not done yet...

apr_status_t apr_brigade_length (apr_bucket_brigade *bb, int read_all, apr_ssize_t *length)

Set length to the total length of bb's buckets. read_all controls the behaviour when a bucket with an unknown length is encountered. If read_all is set, such buckets are read. Otherwise, length is set to -1 and the function reports success.

Misc

int apr_brigade_to_iovec (apr_bucket_brigade *brigade, struct iovec *vec, int nvec)

Fill out the iovec vec with read data from up to nvec buckets in brigade returns the number of elements actually assimilated. This is useful for writing to a file or to the network efficiently.

Brigade macros

Following is a set of macros to manipulate and to tests on a bucket brigade. For brevity and maintenance purposes, it is recommended that you use these macros and not manipulation of the bucket and brigade data structures.

APR_BRIGADE_SENTINEL (brigade)

Bucket brigades are implemented over Apache's ringinterface (srclib/apr-util/include/apr_ring.h). This macro returns the marker for the limits of the ring, ie. the end of brigade. example

APR_BRIGADE_EMPTY (brigade)

Determine if brigade is empty.

APR_BRIGADE_FIRST (brigade)

Return the first bucket in a brigade.

APR_BRIGADE_LAST (brigade)

Return the last bucket in brigade.

APR_BRIGADE_FOREACH (bucket, brigade)

Iterate through a brigade.

APR_BRIGADE_INSERT_HEAD (brigade, bucket)

Insert a list of buckets at the front of a brigade.

APR_BRIGADE_INSERT_TAIL (brigade, bucket)

Insert a list of buckets at the back end of a brigade.

APR_BRIGADE_CONCAT (brigade1, brigade2)

Concatenate brigade2 onto the end of brigade1.

Bucket list manipluation macros

Buckets are srclib/apr-util/include/apr_ring.h (see srclib/apr-util/include/apr_ring.h) so a bucket has pointers to previous and next neighbors. Following is a list of macros for manipulating the list features of buckets. For brevity and maintenance purposes, it is recommended that you use these macros and not manipulation of the bucket and brigade data structures.

APR_BUCKET_INSERT_BEFORE (insertMe, beforeMe)

Insert a list of buckets starting with insertMe before beforeMe.

APR_BUCKET_INSERT_AFTER (insertMe, afterMe)

Insert a list of buckets starting with insertMe after afterMe.

APR_BUCKET_NEXT (bucket)

Get bucket's next neighbor.

APR_BUCKET_PREV (bucket)

Get bucket's previous neighbor.

APR_BUCKET_REMOVE (e)

Remove bucket from its brigade.

APR_BUCKET_INIT (bucket)

Initialize a new bucket's prev/next pointers.

Eric Prud'hommeaux, 23th August 200