Lecture 02 – 10/3/2016

Seite 1

File System Part I

C – API

File table

Keyboard

Screen

cout

Difference between:

cout

<< “Ducks are good” << endl;

cerr

<< “Lizards are cute” << endl;

If you want to really see the cerr there

are ways around that

Preliminaries

Why discuss this first? Because we know something about file

systems, it is a nice simple place to start. Also file access is the foundation for

other services in the UNIX. E.g. pipes and internet communications.

Review the hardware fundamentals. Format of disk, seek, latency,

transfer rate.

Discuss some of the directory structure in UNIX. I would like to

say that it is consistently defined on every system, but it isn't.

/bin

: were most of the commands and system programs may be found.

-> Examples mkdir, ls, ed.

/etc

: Just what the name implies. A place to put a whole bunch of stuff.

-> Examples: passwd and hosts file

/usr

: Usually information associated with users. (i.e. Home directories - we

use /home)

-> Example student and faculty sub-directories.

/usr/include

: where many of the include files may be found. Like iostream

/tmp

: temporary directory. Goes away on bootup.

/unix

: where UNIX system found. We are using /kernel

/dev

: description of devices.

Basic services that we expect any

file system to provide.

create

open

On many file systems, include Linux, the open also serves the role of

create

read

write

close

rewind

seek

delete

Notes on the UNIX file system:

UNIX see a file as a string of bytes. The UNIX

operating system does NOT see records or lines.

Mention that UNIX makes use of disk caching. Files accesses

are not buffered within the code for these calls. Let's discuss the

pros and cons of this in class.

There

are also C and C++ files systems. These are implemented using the UNIX

file system. Why use the UNIX file system rather than the file

system provided in a programming language? Why use the file system

provided by the programming language?

File names may be made up of any characters. For example

"*" and " " may be file names. You can even have

the ASCII code to ring the "bell:. Hard to remove these using

the rm command.

Microsoft

has implemented the basic UNIX file system calls. So, if you ever

need unbuffered I/O these will do it for you. I will explain why in

class.

We access these file services through UNIX system calls. These

look like C function calls, because they are. Is this a problem??

It does require a way to identify an open file.

For example:

nb = read( fd,

buff, NumBytes );

System calls "talk" to the operating system. How does

this happen? (svc, int, sys assembler calls are made to transfer control to the

operating system). Because of requiring a context switch to

and from the operating system, there is additional overhead associated with

calls like the read.

How are errors returned? Rule: UNIX usually returns -1 if there is

an error. For functions that return addresses, NULL is usually

returned if there is an error. The external int errno contains a code

that describes the error. We MUST always check for errors on each system call we

use. I will not accept your lab if you don't do this. Make sure that everyone

understands external variables.

Let's look at errno.h to see what is there. (Look in the

/usr/include directory.) Note the extern int errno; statement.

Note that errno can be a function. Why? Note: the errno.h file

contains other include file. These contain the defines for the error

codes.

Error Reporting

Error reporting is essential. When a function in the API

fails, you want to find this out and why. We will use the perror function

to report errors. Let's look this function up in the man pages. It

is a lot easier to do this on the site:

http://linux.die.net

This function reports the error number, a description of the error

and your message. I expect to see you using it on your quizzes and

programming assignments.

Previous to this function you had to write your own to do

it. Those days are long gone. See below for an example of the type

of function you would have to write. We will talk about it for the sake

of its ideas.

errno contains a number that stands for the error. Sometimes

it is nicer to read text. "sys_errlist" is an array of pointers to

ASCII strings that describe the errors. "sys_nerr" is the size of

this array. These are global variables The following is a function that we will

use to display errors. Note that the man pages list error returns

from system calls. We are discussing this function because it opens up a

discussion ofa number of ideas.

The following is an example of how to write your own function for

reporting errors. I will use perror in the class examples and

expect you to use it in your programming assignments and quizes.

#include <iostream>

using namespace std;

void syserr( const char *msg ) /*

print sys error message and terminate. */

{

extern

int errno, sys_nerr;

extern

char *sys_errlist[];

cerr

<< "ERROR: " << msg << "(" <<

errno;

if(

errno > 0 && errno < sys_nerr ) {

cerr

<< " - " << sys_errlist[errno] << ")"

<< endl;

}

else {

cerr

<< ")" << endl;

}

exit(1);

}

Why cerr instead of cout?

See ex1.cpp for an example of how perror works. I

will use the gdb (our debugger) to demonstrate this function. Also, discuss the

basics of make. See Makefile file for the make file for this

project. We will also create this program in Eclipse.

Note: it is not a good move to name your program test.

Note: for Eclipse, see the following article if there is not an

autosave before a build:

http://help.eclipse.org/juno/index.jsp?topic=%2Forg.eclipse.cdt.doc.user%2Ftasks%2Fcdt_t_autosave.htmhttp://help.eclipse.org/juno/index.jsp?topic=%2Forg.eclipse.cdt.doc.user%2Ftasks%2Fcdt_t_autosave.htm

The following is an example of more sophisticated file use to

get a perspective on why these calls are very useful.

Discuss the problem of accessing minute data from FasTick.

Picture simulations that require 500 days of 1 minute tick data with a

thousands stocks. By using FasTick directly, it would take about 10 hours

to download. So each simulation involving this data would take more than

10 hours. We need an alternative approach. What approach do you think

I came up with

The file access calls 9/26

starts here. Start by creating a workspace

Let's go through the basic system calls to do file access. We are

diving right in rather than given background material first. We will come back

to things like how the file system is set up. Note: Windows

have a similar set of unbuffered calls that you might want to use when you need

this functionality.

The Linux file system is important because large parts of the

operating system use it. For example, process to process communications

and network programming go through the file system.

The first API call presented will be the open. You can

find all the information I am discussing using the man command or by reading

the text or look this up on the web. (Note: the following link is very

useful for man pages:http://linux.die.net/man/ Yes, I am

repeating myself.) I will show how to do this through emacs. The

open call prepares your program to access a file. What does

this mean??

Note: there is an ancient call called creat to

create new files. We will not talk about it, but you might run into

it. It has a similar format to the open function and has beenmade obsolete by

the open call. Someone who uses it has probably been around for a while

or the code you are working on has been.

#include < sys/types.h >

#include < sys/stat.h >

#include < fcntl.h >

#include < limits.h > /* definition of OPEN_MAX */

int open( const char * path, int flags,

mode_t mode)

path : the name of the file optionally including a full

directory path. If no path is given, the file is assume to be in the working

directory from which the program was executed.

flags : defines how the file is to be opened. This argument is

formed by OR'ing the following values (I will not list all of them):

O_RDONLY

: Open for reading only.

O_WRONLY

: Open for writing only.

O_RDWR

: Open for reading and writing.

O_APPEND

: Append on each write.

O_CREAT

: Create file if it does not exist.

O_TRUNC

: Truncate size to 0.

O_EXCL

: Error if create and file exists.

O_SYNC

: Do file writes synchronously. That is each write must

fully complete before and other operation occurs. This guy is super

safe.

Make sure everyone recalls how to do bit wise or's using the

"|" operator. Note: some flags are system dependent. It

is always good to choose the least level of flags. Why?

mode : this parameter is optional. (How does this

happen?) It is only used if the file is created by this call. The mode is used

to describe permissions associated with the file In the old days people used a

3 digit octal number to describe the permission. The digits correspond to

owner, group, and other. The bits within a digit indicate read, write, and

execute permissions. How do you specify an octal number in C++. To

give read and write permission to the owner only, mode = 0600.

Lets look at /usr/include/sys/stat.h for newer way of specifying

permissions. I will stick with the old way. Usually the new way is

better, but at least for me, the old way is quicker to write and almost

everyone understands it. The following is a section from the stat.h

file:

#define S_IRWXU 00700 /* read, write, execute: owner */

#define S_IRUSR 00400 /* read permission: owner */

#define S_IWUSR 00200 /* write permission: owner */

#define S_IXUSR 00100 /* execute permission: owner */

#define S_IRWXG 00070 /* read, write, execute: group */

#define S_IRGRP 00040 /* read permission: group */

#define S_IWGRP 00020 /* write permission: group */

#define S_IXGRP 00010 /* execute permission: group */

#define S_IRWXO 00007 /* read, write, execute: other */

#define S_IROTH 00004 /* read permission: other */

#define S_IWOTH 00002 /* write permission: other */

#define S_IXOTH 00001 /* execute permission: other */

Make sure that if you specify the O_CREAT flag that you put this

option in. Otherwise, life will get interesting. :-) I will explain

why.

Returns -1 if there is and error and a nonnegative integer (file

descriptor) if the function succeeds. All subsequent system calls that

reference this file will use this file descriptor.

Notes:

File descriptors 0, 1, and 2 are are pre-assigned. These correspond

to the keyboards, screen, and screen (for error messages)

respectively. If you close file descriptor 1, you will cause

interesting effects. For example, the cout object will no longer

work. Why?

The

previous statement has interesting implications. It means that

access to a screen or from a keyboard goes through file descriptors in the

same manner as a file. This is also true of other means of

communications and other devices.

File

descriptors represent a position in a local table.

File

descriptors are not global across processes. That means file

descriptor 4 in one process has no relationship to file descriptor 4 in

another.

File

descriptors are assigned lowest available number first. So, on your

first open, your should see a File descriptor having the value of 3.

There

is a limit on how many files may be opened by a single process. Most

Linux systems have 1024 as the limit. How would you find this

out? (ulimit -n) This can be changed.

If

you use the equivalent calls in Microsoft OSs, be aware that the file

descriptors are not assigned sequentially and are not small numbers.

There

is a separate function to create files called creat. This

function is less powerful than open and obsolete, so it should not be

used. It exists for the sake of maintaining old code.

The read function reads data from an open file. It has the

following format:

ssize_t read(int fd, void *buff, size_t

nbyte);

fd : the file descriptor for an open file.

nbytes : the number of bytes to be read.

buff : address of a buffer into which the data will be read.

What is size_t? What is ssize_t. What the heck are they

thinking!

Returns: If the returned value is 0, then end-of-file has been

reached. If the read is successful, the number of bytes actually read is

returned. Otherwise, a -1 is returned and the global variable errno is set to

indicate the error.

Note about where in the file we are reading from and what the

"current file" offset or file pointer refers to.

Note: man page can describe the arguments slightly differently

depending on whose UNIX we are using. But, the use is the same.

This is one of the problems with using UNIX.

The write system call is used to write data to a file. It has the

following format:

ssize_t write(int fd, const void *buff,

size_t nbytes);

fd : the file descriptor for an open file.

nbytes : the number of bytes to be written.

buff : buffer for the data to be written.

This function returns the number of bytes written to the file. It returns -1 if

there was an error. Question: why the use of const?

The close function allows the operating system to

recover the resources that were allocated when the file was opened. All the

files that your program has opened are automatically closed when your program

terminates. The close function has the following format:

int close(int fd)

fd : the file descriptor of an open file.

This function returns 0 if it is successful and -1 if it fails.

Sometime you might want to read from the middle of a file.

Why? The lseek function allows you to position the current file offset so

that you can read from or write to anywhere in the file. If you try to read

beyond the end of the file what do you think happens? You get a zero length

read. If you attempt to write beyond the end of a file what do you think

happens?

The format of the lseek is as follows:

#include < sys/types.h >

#include < unistd.h >

off_t lseek( int fd, off_t offset, int

whence)

fd: a file descriptor from an open file.

offset: a file offset.

whence: indicates what the offset is from.

If whence is SEEK_SET, the pointer is set to offset bytes.

If whence is SEEK_CUR the pointer is set to its current location plus offset.

If whence is SEEK_END, the pointer is set to the size of the file plus the

offset.

Note: the offset may be positive or negative.

Note: lseek can be used to find the current location. How?

Returns the new file offset if successful and -1 if fails.

Examples:

ex2.cpp and copy.cpp -

is an example of a function to copy the contents of one file into

another. I have an error reporting function. We will replace

it with perror. Note the way I can use the debugger to find why a program

core dumps. We will also create this executable without using Eclipse.

ex3.cpp Shows the use of lseek and what happens when you write

beyond the end of a file. Note about how this can be used to mess up

people who do backups. :-) DO NOT DO THIS!

Lab1

Write a program called rctail which will take a file name as its command line

argument and display the last 10 lines in the file. Don't make any assumptions

on the size of the file you are displaying. You are to use the UNIX system

calls. Make this program as efficient as

possible. We will discuss more in class. To get it right,

this lab takes more time that you would expect, so start it right away.

Note: In class we will determine how lines of text in a file

are delimited. We will use the od command. This allows us to dump

in octal and other formals.

If there is time, write an example in class that will display the

contents of a file in reverse order.

I/O Efficiency

Recall that disk data is cached. So, this does not mean we

have to read from the disk every time we want data. The text has a nice

table (3.5) that I would like to discuss. It give measures of speed in

accessing data from a large file through Linux. It uses a variety of

buffer sizes. Please read the text about this table and how they did the

experiment. Note, they are doing fresh reads from the disk. Why?

Buff Size

User CPU Time

System CPU Time

Clock Time

#Loops

1

124.89

161.65

288.64

103,316,352

2

63.10

80.96

145.81

51,658,176

128

1.01

1.27

6.82

807,159

1024

0.17

0.23

7.84

100,894

4096

0.03

0.16

6.86

25,223

16,384

0.02

0.18

6.87

6,305

524,288

0.00

0.22

7.35

198

Note:

Reads and writes are atomic. That is they will not be

interrupted once started. That is a write to a file will be allowed to

complete before another read or write operation can occur. *How is this

accomplished by the operating system?) This is important for

multithreaded and multiprocess applications.

The pread and pwrite function

calls

These calls combine seek and a read or write in a single

operation. Why???

The readv and writev function

calls

These calls allow us to read and write vectors of data.

Why?? Maybe later.