Playing Around With Linux / POSIX File Descriptors in C
posts Linux POSIX File Descriptors I/O Redirection Interprocess Communication IPC Pipes FIFOs Concurrency Parallelism Resource Management Error Handling POSIX Standards BashWas helping out a bit with this PR where one user was trying to get rid of special token being outputted in llama.cpp when being run in shell script context. One approach I thought was to allow for outputting the control tokens into a separate out of band stream rather than mixed into stdout.
Ultimately we found a different approach that didn't require attaching an out of band stream approach.
Nevertheless, I'm interested in understanding more about the non-standard pipes and how to use it in linux. Which would allow for multiple pipe output and input of a program besides stdout and stdin.
Within C standard library FILE
stream add buffering and portability for text and binary I/O,
for ease of use in standard C programs and by default on program initialization stdin, stdout
and stderr FILE
stream is opened and associated with either a File Descriptor (Linux/Unix)
or a HANDLE
for Windows.
File descriptors on the other side provides direct low level access to system resources like files, pipes and sockets.
The key thing to note is that this is purely a linux facility and windows doesn't have this capability as this write up is focused on utilizing File descriptor. Instead Windows reply on
These File Descriptors can be interacted using system calls provided by unistd.h
library's
pipe()
, fork()
and other Unix-centric IPC mechanism.
Unix File Descriptor | Windows API HANDLE | Standard Stream |
---|---|---|
0 | GetStdHandle(STD_INPUT_HANDLE) | Standard Input |
1 | GetStdHandle(STD_OUTPUT_HANDLE) | Standard Output |
2 | GetStdHandle(STD_ERROR_HANDLE) | Standard Error |
3 | 3 | - |
4 | 4 | - |
etc... | etc... | - |
Experimenting With Pipes
These C test program was run under Linux Mint 21.3 x86_64
.
Print Out Attachments Status
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
int main() {
for (int i = 0; i < 10; i++) {
printf("Output descriptor %i is %s\n", i, (fcntl(i, F_GETFL) == -1) ? "Not Attached" : "Attached");
}
return 0;
}
Output on bash console:
$tcc -run ATTACHCHECK.c
Output descriptor 0 is Attached
Output descriptor 1 is Attached
Output descriptor 2 is Attached
Output descriptor 3 is Not Attached
Output descriptor 4 is Not Attached
Output descriptor 5 is Not Attached
Output descriptor 6 is Not Attached
Output descriptor 7 is Not Attached
Output descriptor 8 is Not Attached
Output descriptor 9 is Not Attached
$tcc -run ATTACHCHECK.c 3>/dev/null 6>/dev/null
Output descriptor 0 is Attached
Output descriptor 1 is Attached
Output descriptor 2 is Attached
Output descriptor 3 is Attached
Output descriptor 4 is Not Attached
Output descriptor 5 is Not Attached
Output descriptor 6 is Attached
Output descriptor 7 is Not Attached
Output descriptor 8 is Not Attached
Output descriptor 9 is Not Attached
Single Attach Test
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
int main() {
printf("Output descriptor 3 is %s\n", (fcntl(3, F_GETFL) == -1) ? "Not Attached" : "Attached");
return 0;
}
Output:
$ tcc -run ATTACHTEST.c 3>a
Output descriptor 3 is Attached
$ tcc -run ATTACHTEST.c
Output descriptor 3 is Not Attached
Testing File Descriptor 0 behavior
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main() {
char outbuff[1000] = {0};
size_t outbuff_length = 0;
FILE *file_f = fdopen(0, "r");
if (file_f) {
char *buff = NULL;
char inbuff[1000] = {0};
ssize_t nread = read(0, inbuff, sizeof(inbuff) - 1);
// Remove newline character if present
if (nread > 0 && inbuff[nread - 1] == '\n') {
inbuff[nread - 1] = '\0';
nread--; // Decrement nread to exclude the newline character
}
if (nread != -1) {
outbuff_length = snprintf(outbuff, sizeof(outbuff), "Tried to read 0 got '%s' (%ld)\n", inbuff, nread);
} else {
outbuff_length = snprintf(outbuff, sizeof(outbuff), "Tried to read 0 but got no input\n");
}
free(buff);
} else {
outbuff_length = snprintf(outbuff, sizeof(outbuff), "Tried to read 0 but cannot open file descriptor\n");
}
write(0, outbuff, outbuff_length);
write(1, outbuff, outbuff_length);
return 0;
}
When run directly the program is able to read the piped in 'hello' as expected at File Descriptor 0.
$ echo "hello" | tcc -run INOUT.c
Tried to read 0 got 'hello' (5)
Piping in to File Descriptor 0 via a FIFO pipe works as expected as well.
$ mkfifo fd0
$ echo "input_0" > fd0 &
[1] 373014
$ tcc -run INOUT.c 0<fd0
Tried to read 0 got 'input_0' (7)
[1]+ Done echo "input_0" > fd0
However if you try to pipe out data from File Descriptor 0... it will hang as the read() system call fails/blocks indefinitely because there is no valid data source for stdin.
$ mkfifo fd0
$ echo "input_0" > fd0 &
[1] 376622
$ tcc -run INOUT.c 0>fd0
... Program Crashes Here ...
Also bash will throw an error if you try to output File Descriptor 0 to a file.
I suspect 0>a
did open as an output so hence the program cannot read from
the piped in "hello". Which means bash does not prevent me from redirecting 0 to
a file but this breaks the program logic.
$ echo "hello" | tcc -run INOUT.c 0>a
Tried to read 0 but cannot open file descriptor
Testing non standard output stream on File Description 0
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main() {
write(0, "hello\n", 7);
return 0;
}
Output:
$ echo "yellow" | tcc -run OUTTOD0.c 0>a
$ cat a
hello
This indicates that it's possible to do non standard output
Detecting Attachment State
#include <stdio.h>
#if defined (__unix__) || (defined (__APPLE__) && defined (__MACH__))
#include <unistd.h>
#elif defined(_WIN32)
#include <io.h>
#endif
int main() {
char name[1000] = {0};
scanf("%s", &name);
printf("hello %s %s\n", name, isatty(fileno(stdin)) ? "you decided to come in via terminal?" : "you decided to come in via pipe?");
return 0;
}
Output via pipe:
$ tcc -run stdinattach.c << __HEREDOC__
$ mark
$ __HEREDOC__
hello mark you decided to come in via pipe?
Output when manually called via interactive terminal:
$ tcc -run stdinattach.c # Then typed in yola manually
hello yola you decided to come in via terminal?
Anyway I hope all these examples above gives you a better sense of how this works.