-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
buffer: refactor buffer to be dynamically-sized #53
base: master
Are you sure you want to change the base?
Changes from 1 commit
8932c77
14a9563
3cf9285
e76558a
a36af8d
5ee2021
154d1a3
6e9c709
ef706f9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -79,8 +79,16 @@ mpd_async_new(int fd) | |
async->fd = fd; | ||
mpd_error_init(&async->error); | ||
|
||
mpd_buffer_init(&async->input); | ||
mpd_buffer_init(&async->output); | ||
if (mpd_buffer_init(&async->input) == NULL) { | ||
async->output.data = NULL; | ||
mpd_async_free(async); | ||
return NULL; | ||
} | ||
|
||
if (mpd_buffer_init(&async->output) == NULL) { | ||
mpd_async_free(async); | ||
return NULL; | ||
} | ||
|
||
return async; | ||
} | ||
|
@@ -92,6 +100,8 @@ mpd_async_free(struct mpd_async *async) | |
|
||
mpd_socket_close(async->fd); | ||
mpd_error_deinit(&async->error); | ||
mpd_buffer_end(&async->input); | ||
mpd_buffer_end(&async->output); | ||
free(async); | ||
} | ||
|
||
|
@@ -179,14 +189,17 @@ mpd_async_read(struct mpd_async *async) | |
{ | ||
size_t room; | ||
ssize_t nbytes; | ||
const size_t min_size = 256; | ||
|
||
assert(async != NULL); | ||
assert(async->fd != MPD_INVALID_SOCKET); | ||
assert(!mpd_error_is_defined(&async->error)); | ||
|
||
if (!mpd_buffer_make_room(&async->input, min_size)) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What if you're receiving data from a server which pushes a terabyte to you? DoS bug. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is my understanding, maybe i'm missing something.
For (i), you will increase the buffer by 256 bytes (if needed) and read data out. If the message was larger than 256 bytes, an error will occur in For (ii), the flush/write will stop once output data is no longer available. Output data is generated locally. Assuming you wrote 32MB of output data [1] (13 loops on You can increase the input buffer size by calling [1] MPD will not accept this, but lets say it does for this example. If you prefer, we could provide in the meson configure the options for the first and limit allocation sizes for the buffer. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think you're wrong. You assume that the message will be delivered in chunks of 256 bytes, but why do you believe that is true? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see two cases if you send a single 1 TB line (ii) You can ask
|
||
mpd_error_code(&async->error, MPD_ERROR_OOM); | ||
return false; | ||
} | ||
room = mpd_buffer_room(&async->input); | ||
if (room == 0) | ||
return true; | ||
|
||
nbytes = recv(async->fd, mpd_buffer_write(&async->input), room, | ||
MSG_DONTWAIT); | ||
|
@@ -276,39 +289,25 @@ mpd_async_io(struct mpd_async *async, enum mpd_async_event events) | |
return true; | ||
} | ||
|
||
bool | ||
mpd_async_send_command_v(struct mpd_async *async, const char *command, | ||
va_list args) | ||
static bool | ||
quote_vargs(struct mpd_buffer *buffer, char *start, char **end_pos, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What does this function do? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed in 14a9563 |
||
va_list args) | ||
{ | ||
size_t room, length; | ||
char *dest, *end, *p; | ||
char *end, *p; | ||
const char *arg; | ||
|
||
assert(async != NULL); | ||
assert(command != NULL); | ||
|
||
if (mpd_error_is_defined(&async->error)) | ||
return false; | ||
|
||
room = mpd_buffer_room(&async->output); | ||
length = strlen(command); | ||
if (room <= length) | ||
return false; | ||
|
||
dest = mpd_buffer_write(&async->output); | ||
/* -1 because we reserve space for the \n character */ | ||
end = dest + room - 1; | ||
|
||
/* copy the command (no quoting, we asumme it is "clean") */ | ||
|
||
memcpy(dest, command, length); | ||
p = dest + length; | ||
/** | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Don't use Doxygen-style comments unless you're documenting an API. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed in 14a9563 |
||
* mpd_async_send_command_v() guarantees that mpd_buffer_room() has at | ||
* least 1 position available (length + 1) | ||
*/ | ||
assert(mpd_buffer_room(buffer) >= 1); | ||
end = start + mpd_buffer_room(buffer) - 1; | ||
p = start; | ||
|
||
/* now append all arguments (quoted) */ | ||
|
||
while ((arg = va_arg(args, const char *)) != NULL) { | ||
/* append a space separator */ | ||
|
||
if (p >= end) | ||
return false; | ||
|
||
|
@@ -317,17 +316,57 @@ mpd_async_send_command_v(struct mpd_async *async, const char *command, | |
/* quote the argument into the destination buffer */ | ||
|
||
p = quote(p, end, arg); | ||
assert(p == NULL || (p >= dest && p <= end)); | ||
assert(p == NULL || (p >= start && p <= end)); | ||
if (p == NULL) | ||
return false; | ||
} | ||
|
||
|
||
/* append the newline to finish this command */ | ||
|
||
*p++ = '\n'; | ||
*end_pos = p; | ||
return true; | ||
} | ||
|
||
bool | ||
mpd_async_send_command_v(struct mpd_async *async, const char *command, | ||
va_list args) | ||
{ | ||
char *end_pos = NULL, *write_p; | ||
bool success = false; | ||
size_t length; | ||
va_list cargs; | ||
|
||
assert(async != NULL); | ||
assert(command != NULL); | ||
|
||
if (mpd_error_is_defined(&async->error)) | ||
return false; | ||
|
||
length = strlen(command); | ||
/* we need a '\n' at the end */ | ||
if (!mpd_buffer_make_room(&async->output, length + 1)) { | ||
mpd_error_code(&async->error, MPD_ERROR_OOM); | ||
return false; | ||
} | ||
|
||
/* copy the command (no quoting, we assume it is "clean") */ | ||
memcpy(mpd_buffer_write(&async->output), command, length); | ||
mpd_buffer_expand(&async->output, length); | ||
|
||
while (!success) { | ||
va_copy(cargs, args); | ||
write_p = mpd_buffer_write(&async->output); | ||
success = quote_vargs(&async->output, write_p, &end_pos, cargs); | ||
va_end(cargs); | ||
if (!success) { | ||
if (!mpd_buffer_double_buffer_size(&async->output)) { | ||
mpd_error_code(&async->error, MPD_ERROR_OOM); | ||
return false; | ||
} | ||
} | ||
} | ||
|
||
mpd_buffer_expand(&async->output, p - dest); | ||
mpd_buffer_expand(&async->output, end_pos - write_p); | ||
return true; | ||
} | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -32,10 +32,10 @@ | |
#include <assert.h> | ||
#include <string.h> | ||
#include <stdbool.h> | ||
#include <stdlib.h> | ||
|
||
/** | ||
* A fixed 4kB buffer which can be appended at the end, and consumed | ||
* at the beginning. | ||
* A buffer which can be appended at the end, and consumed at the beginning. | ||
*/ | ||
struct mpd_buffer { | ||
/** the next buffer position to write to */ | ||
|
@@ -45,17 +45,37 @@ struct mpd_buffer { | |
unsigned read; | ||
|
||
/** the actual buffer */ | ||
unsigned char data[4096]; | ||
unsigned char *data; | ||
|
||
/** size of buffer */ | ||
size_t data_len; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Badly named variable. This is not the "length" of "data". There is rarely ever that much data in the buffer! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You are right. I've renamed to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. But that's still a bad name. Your data is not this size. You don't have that much data. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we call it There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's still ambiguous - reading such a name, I'd think that's the configured limit. What about There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. changed in ef706f9 |
||
}; | ||
|
||
/** | ||
* Initialize an empty buffer. | ||
*/ | ||
static inline void | ||
static inline void * | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Undocumented return value. Why would a caller ever be interested in the value of the allocated buffer pointer? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This can be changed to bool. The idea is to check if the buffer was able to be allocated There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If that's your idea, then your return value should express exactly that. And should be documented. |
||
mpd_buffer_init(struct mpd_buffer *buffer) | ||
{ | ||
const size_t std_data_len = 4096; | ||
|
||
buffer->read = 0; | ||
buffer->write = 0; | ||
buffer->data_len = std_data_len; | ||
buffer->data = malloc(buffer->data_len); | ||
if (buffer->data != NULL) | ||
memset(buffer->data, 0, buffer->data_len); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is not necessary; it is just a precaution to initialize the data to a known state |
||
|
||
return buffer->data; | ||
} | ||
|
||
/** | ||
* Free the buffer area. | ||
*/ | ||
static inline void | ||
mpd_buffer_end(struct mpd_buffer *buffer) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd prefer There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No problem. Fixed in e76558a |
||
{ | ||
free(buffer->data); | ||
} | ||
|
||
/** | ||
|
@@ -79,12 +99,13 @@ mpd_buffer_move(struct mpd_buffer *buffer) | |
static inline size_t | ||
mpd_buffer_room(const struct mpd_buffer *buffer) | ||
{ | ||
assert(buffer->write <= sizeof(buffer->data)); | ||
assert(buffer->write <= buffer->data_len); | ||
assert(buffer->read <= buffer->write); | ||
|
||
return sizeof(buffer->data) - (buffer->write - buffer->read); | ||
return buffer->data_len - (buffer->write - buffer->read); | ||
} | ||
|
||
|
||
/** | ||
* Checks if the buffer is full, i.e. nothing can be written. | ||
*/ | ||
|
@@ -125,7 +146,7 @@ mpd_buffer_expand(struct mpd_buffer *buffer, size_t nbytes) | |
static inline size_t | ||
mpd_buffer_size(const struct mpd_buffer *buffer) | ||
{ | ||
assert(buffer->write <= sizeof(buffer->data)); | ||
assert(buffer->write <= buffer->data_len); | ||
assert(buffer->read <= buffer->write); | ||
|
||
return buffer->write - buffer->read; | ||
|
@@ -154,4 +175,50 @@ mpd_buffer_consume(struct mpd_buffer *buffer, size_t nbytes) | |
buffer->read += nbytes; | ||
} | ||
|
||
/** | ||
* Makes at least min_size bytes available for writing data. | ||
* In other words, mpd_buffer_room() will be >= min_size if called after this | ||
* function. | ||
*/ | ||
static inline bool | ||
mpd_buffer_make_room(struct mpd_buffer *buffer, size_t min_avail_len) | ||
{ | ||
size_t newsize; | ||
|
||
if (mpd_buffer_room(buffer) >= min_avail_len) | ||
return true; | ||
|
||
newsize = buffer->data_len * 2; | ||
while (newsize < min_avail_len) | ||
newsize *= 2; | ||
|
||
/* empty buffer */ | ||
if (mpd_buffer_room(buffer) == buffer->data_len) { | ||
free(buffer->data); | ||
buffer->data = malloc(newsize); | ||
if (buffer->data == NULL) | ||
return false; | ||
memset(buffer->data, 0, newsize); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For the same reason it is used in |
||
} else { | ||
buffer->data = realloc(buffer->data, newsize); | ||
if (buffer->data == NULL) | ||
return false; | ||
|
||
/* clear region not committed and new region */ | ||
memset(mpd_buffer_write(buffer), 0, | ||
newsize - (buffer->data_len - mpd_buffer_room(buffer))); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Remove stale (non committed) data from the buffer There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't understand. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is to keep the data in a known state. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. But what's the point of that? Why is "zero state" any more special than "random uncommitted data"? Zero is an arbitrary value just like any other. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. okay, I will remove the code related to the known state |
||
} | ||
buffer->data_len = newsize; | ||
return true; | ||
} | ||
|
||
/** | ||
* Double the buffer area. | ||
*/ | ||
static inline bool | ||
mpd_buffer_double_buffer_size(struct mpd_buffer *buffer) | ||
{ | ||
return mpd_buffer_make_room(buffer, buffer->data_len * 2); | ||
} | ||
|
||
#endif |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -100,27 +100,7 @@ bool | |
mpd_sync_send_command_v(struct mpd_async *async, const struct timeval *tv0, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. PS: struct timeval is not used here because `mpd_sync_io' is not called anymore; There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since this is an internal API, we can remove the parameter easily. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The refactoring is to allow libmpdclient to be limited in its message size only by MPD. If i'm right, currently MPD supports up to 8KiB messages. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You didn't answer my question. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The reasons were: (i) At least one user would like to send messages > 4KiB (issue #51), and (ii) this code would also adapt in case MPD changes the limit of received messages. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. #51 does not request being able to send large requests. He complains about the lousy error handling if he tries to send a huge malformed request. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. From the debian bug report: I think for the majority of cases 4KiB is more than sufficient; The reason i submitted the other PR is in case the changes i'm proposing here are deemed undesirable There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. True, that's where the Debian bug report deviates from #51, but we were talking about #51. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. He also mentions this feature request in #51. The title is about the hard coded limit. True, the test seems to be about fuzzing, but from the MPD protocol page i didn't find any mention of a limit to the find/search command. In theory, you could concatenate multiple find requests in a single huge line like: |
||
const char *command, va_list args) | ||
{ | ||
struct timeval tv, *tvp; | ||
va_list copy; | ||
bool success; | ||
|
||
if (tv0 != NULL) { | ||
tv = *tv0; | ||
tvp = &tv; | ||
} else | ||
tvp = NULL; | ||
|
||
while (true) { | ||
va_copy(copy, args); | ||
success = mpd_async_send_command_v(async, command, copy); | ||
va_end(copy); | ||
|
||
if (success) | ||
return true; | ||
|
||
if (!mpd_sync_io(async, tvp)) | ||
return false; | ||
} | ||
return mpd_async_send_command_v(async, command, args); | ||
} | ||
|
||
bool | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an obscure way of fake-initializing the buffer to allow
mpd_async_free()
(thempd_buffer
API doesn't document that this is legal). I don't like obscure code like that.If you refactor this to never fail (like before) and do the first allocation lazily, you can avoid this fragile and cumbersome code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case, we would need to do the allocation in
mpd_buffer_room()
; currently, it returnssize_t
which would make the OOM case awkward (return(size_t)-1
for instance). We could change that function to write the size in a pointer. Would you be okay with any of these solutions?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would
mpd_buffer_room()
need to do the allocation? It is a simple read-only function which gives information about the buffer state.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The idea was to initialize in a single place; i realize now that read functions does not use the
mpd_buffer_room()
I will update the code to allocate in
mpd_buffer_read()
andmpd_buffer_write()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated code in commit 154d1a3.
This resulted in an increase of code complexity as the code checks for OOM events in more places.