Using JFFS2

Name

JFFS2 usage -- Description of how to use JFFS2

Mounting

A JFFS2 filesystem can be mounted just like any normal eCos filesystem, using the mount() function from the POSIX file I/O package (CYGPKG_FILEIO). You must choose an appropriate Flash I/O block device to use. Documentation on Flash I/O block devices can be found in the Generic Flash package documentation.

Example 1. Mounting and unmounting a JFFS2 filesystem

#include <cyg/fileio/fileio.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
    …
    int rc;
    rc = mount( "/dev/flash/fis/jffs2test", "/fs", "jffs2" );
    if (rc < 0)
       printf( "mount returned error: %s\n", strerror(errno) );
    …
    rc = umount( "/fs" );
    if (rc < 0)
       printf( "umount returned error: %s\n", strerror(errno) );
    …
       

No file system needs to be created in advance for JFFS2. A new file system image will be instantiated if JFFS2 is pointed at an erased Flash area. Similarly if JFFS2 is pointed at a non-erased Flash area that does not contain valid JFFS2 markers, it will refuse to mount it to prevent destruction of data.

If the Flash device supports locking, you should ensure that the flash region for the JFFS2 filesystem is unlocked before mounting, otherwise it will not be possible to write to the Flash. Alternatively, enabled the CDL option CYGSEM_FS_JFFS2_UNLOCK_FLASH to unlock Flash on mounting. This option is disabled by default for safety reasons — if Flash has been locked, it may be for an important reason.

If mounting an existing JFFS2 filesystem, the mount procedure will search for any unused blocks that have not already been erased, and erase them. This can result in an extended mount time, and so this feature can be disabled with the CDL option CYGOPT_FS_JFFS2_ERASE_PENDING_ON_MOUNT.

Normally it is sufficient to prepare a clean JFFS2 partition as above, and load files into it using RedBoot or an eCos application. But if you do wish to use a pregenerated file system image generated on your host PC, there is a utility named mkfs.jffs2 which can be used to generate an image. On Ubuntu, Fedora and Red Hat-based distributions of Linux it can be found in a package named mtd-utils; in Windows with Cygwin, in the mtd package. Be sure to use the -l or -b options to select the endianness so it corresponds with your target, and you may need to set other options according to the requirements of your Flash hardware such as erase block size. The erase block size to use on the mkfs.jffs2 command line should be the largest flash block size used by your JFFS2 partition on the flash device. Note that the -s or --pagesize options are unrelated to the flash block size, and instead control JFFS2's view of the page size - see here for information on changing the value. Use the --help option for a complete list of mkfs.jffs2 parameters.

Note that JFFS2's memory requirements are not static, and so they may increase over time before stabilising. Larger Flash partitions may require non-trivial amounts of memory, especially at mount time. Memory use may be controlled by removing features such as compression, or by constraining the size of the Flash partition. Configuration options controlling optional features may be found in the JFFS2 package CDL configuration.

JFFS2 has built-in tolerance of Flash errors when erasing and should adapt and work around bit errors that arise as Flash reaches the end of its working life. Obviously this comes at the expense of device capacity.

Garbage collection

By default, JFFS2 performs garbage collection on an as-needed basis. This means that when there are insufficient spare clean flash blocks remaining, JFFS2 will perform repeated garbage collection until a block can be erased and then used for future writes.

Garbage collection involves scanning a flash block and determining which nodes are still used for valid data and which are obsolete - valid data is then relocated to a new alternative block, until only obsolete data remains, at which point the block can be erased. The algorithms which choose which block is nominated for garbage collection do so to ensure wear levelling over the life of the flash device. Nodes representing individual file fragments are able to be coalesced and merged with adjacent file fragments leading to reduced flash use overall due to eliminating some of the overhead caused by node metadata on the Flash. There will also be a consequent simplification in the internal filesystem data structures, resulting in reduced memory consumption and fewer data structures which JFFS2 needs to trawl through when locating data. It will also greatly augment the benefit of compression: when compressing, nodes are considered in isolation, and so small nodes are unlikely to compress well, whereas larger coalesced nodes are more likely to occupy less flash space.

Garbage collection thread

As a result of only performing garbage collection when needed, it can mean that individual writes to files may occasionally take a lengthy period of time to run if a new flash block is required - a whole flash block may need to be garbage collected from scratch, have its live data written to a new block, and be erased, the latter in particular being a very lengthy procedure. As such, JFFS2 offers the option of using a garbage collection thread, which can run in the background to advance the garbage collection process when the filesystem is not otherwise being used.

Of course if a filesystem is going to change too rapidly, or the application is CPU bound by higher priority threads, then the garbage collection thread may not be able to keep up. But for the majority of applications, it can considerably reduce or even virtually eliminate the delays caused by the requirement for occasional garbage collection.

The garbage collection thread can be enabled in the JFFS2 package's configuration using the "Garbage collection background thread" CDL option (CYGOPT_FS_JFFS2_GC_THREAD). The priority of that thread defaults to the lowest possible - 1 above that of the idle thread, but this can be changed with the CYGNUM_JFFS2_GC_THREAD_PRIORITY option. The thread of course requires a stack for execution, the value of which can be optimised with the CYGNUM_JFFS2_GC_THREAD_STACK_SIZE option. Note that one garbage collection thread is started for each mounted JFFS2 filesystem, and so one thread stack is allocated for each.

Usually the garbage collection thread will gradually garbage collect nodes from erase blocks until the block is completely unused by valid data. However it would not actually erase it, leaving that to the usual code path that erases blocks only at the point they are needed. This is because otherwise the Flash system may get locked for the duration of the erase process, preventing any threads, including high priority threads, access to it at that time. Yet the garbage collection thread is intended to be a low priority background thread. Nevertheless, enabling the option CYGSEM_JFFS2_GC_THREAD_CAN_ERASE allows the garbage collection thread to erase blocks as well, if blocks are available for erasure.

Finally, the garbage collection thread may run continuously but does not have to run constantly. By default, the thread will be inactive unless the amount of free space in the filesystem is beginning to run low. The filesystem code makes a judgement based on a myriad of factors to decide when to activate this background GC thread (although its low thread priority may result in it not running in practice, so choose its thread priority wisely in relation to the other running threads). Once active, the garbage collection thread will run until sufficient free space is now available, even if more garbage collection may be possible.

As a variation on this behaviour, it is possible to periodically wake the GC thread to advance garbage collection even when the available free space is not yet low; thus preventing it getting close to running out in the first place. This feature is controlled with the CYGNUM_JFFS2_GC_THREAD_TICKS configuration option, which gives the number of ticks between garbage collection passes, even when space is not low.

How ticks corresponds to real time is unspecified and depends on the HAL and kernel clock configuration (although most ports have traditionally defaulted to 10ms ticks). The value of this option can even be set to 0. Note though, that garbage collection is never considered "done" - the thread will run continuously until the filesystem is unmounted, therefore making it run too frequently may in fact cause unnecessary flash operations, increasing flash wear (even though it will be levelled wear). The value should be chosen according to the expected write pattern.

The number of ticks between garbage collection passes can also be set at runtime, by invoking a cyg_fs_setinfo() call on a filesystem object. There is a corresponding cyg_fs_getinfo() call to retrieve the current delay ticks. For example:

#include <cyg/fs/jffs2/jffs2.h>
  …
   {
     int err;
     cyg_tick_count_t old_delay_ticks;
     cyg_tick_count_t new_delay_ticks;

     err = cyg_fs_getinfo("/jffs2", FS_INFO_JFFS2_GET_GC_THREAD_TICKS, &old_delay_ticks, sizeof(cyg_tick_count));
     assert(err == 0);
     new_delay_ticks = old_delay_ticks*10; // slow down GC
     err = cyg_fs_setinfo("/jffs2", FS_INFO_JFFS2_SET_GC_THREAD_TICKS, &new_delay_ticks, sizeof(cyg_tick_count));
     assert(err == 0);
   }
          

One application of dynamically setting the wakeup delay for the garbage collection thread is so that the thread is more active in times of relative system activity, but operates more slowly when quiescent. This may improve flash life, while still retaining benefits of the garbage collection thread.

The value of the wakeup delay can be set to a special value of (cyg_tick_count_t)-1, which indicates that the thread should not wake up periodically any more, but only when space is running low (the behaviour when CYGNUM_JFFS2_GC_THREAD_TICKS is disabled). Similarly retrieving the current wakeup tick value with cyg_fs_getinfo() may return this special value.

Efficiency

Write size

JFFS2 does not guarantee 100% optimal use of Flash space due to its journalling nature, and the granularity of Flash blocks. It is possible for it to fill up even when not all file space appears to have been used, especially if files have had many small operations performed on them and the Flash partition is small compared to the size of the Flash blocks. It is strongly recommended to have at least 5 or 6 Flash blocks spare, over and above space requirements for data, in order to allow the JFFS2 garbage collector to operate.

It is certainly the case that JFFS2 will work very inefficiently if using many small writes. Unless and until garbage collected, each write will occupy its own JFFS2 node on Flash, and so will incur overhead from the node header. A filesystem is likely to "fill" quickly if written a few bytes at a time, rather than in large chunks, so caution in advised, and more space reserved if that write pattern is anticipated. A common application that can do this is logging. Small writes can also remove the benefits of compression, as there is too little data to compress effectively.

Garbage collection

Use of the garbage collection thread will provide a level of continuous garbage collection. As indicated above, garbage collection can reduce flash usage and memory consumption, and improve performance. Therefore doing so on a continuous basis is a wise idea.

Extra spare space is required in order to allow the JFFS2 garbage collector to operate, over and above space requirements for data. A rule of thumb is to use the following formula:

  Recommended overhead == 2 + (( flashsize/50 ) + (flashsectors*100) + (sectorsize-1)) / sectorsize 
So for example, a 16Mbyte flash with 64Kbyte blocks given over entirely to JFFS2 would actually require an overhead of 6 blocks. Or to look at it another way, trying to write data when you have used up all but 6 blocks worth may result in an ENOSPC error being reported. Due to metadata and write characteristics (e.g. lots of 1 byte writes) it's not possible to easily calculate what that actually translates to in terms of maximum file size. Even the above formula is only a rule of thumb, and it has not been proven to be guaranteed to work in all circumstances. It is recommended to be conservative if possible.

Compression

JFFS2 will default to trying to compress files. However, it may be more memory efficient to disable JFFS2 compression entirely in the CDL configuration, and instead ensure that images are stored compressed when they are downloaded, and use the standard RedBoot mechanism to decompress the files upon loading.

Maximum data node size

By default JFFS2 will operate on chunks of files up to 4 kilobytes in size, but larger chunks may be able to be compressed more efficiently, and have lower metadata overheads. To increase the size, you must change JFFS2's view of the machine page size - the eCos JFFS2 port's view of the page size does not actually need to reflect any real underlying page size of the memory management system, and the notion of the page size is a hangover from the Linux origins of JFFS2 which would be too disruptive to remove. Changing the page size can be performed by changing the page size exponent configuration option (CYGNUM_LINUX_COMPAT_PAGE_SIZE_EXPONENT) in the Linux compatibility layer package (CYGPKG_LINUX_COMPAT). A value of 12 indicates 212 which is 4 kilobytes. For example this could be changed to 16, corresponding to 216 which is 64 kilobytes. If using mkfs.jffs2, make sure that its value for the page size, using the -s or --pagesize options, is the same or lower than the page size given by CYGNUM_LINUX_COMPAT_PAGE_SIZE_EXPONENT.

Configuration dependencies

JFFS2 has a number of package dependencies. As such it may be helpful to use the below eCos minimal configuration (.ecm) file and import it into your configuration to satisfy most dependencies quickly without conflict. This minimal configuration file is usable for building both eCos and RedBoot with JFFS2 included. Note you may need to modify the package versions from current to the version of your release, e.g. v2_0_64.

cdl_configuration eCos {
    package CYGPKG_IO_FLASH current ;
    package CYGPKG_MEMALLOC current ;
    package CYGPKG_COMPRESS_ZLIB current ;
    package CYGPKG_IO_FILEIO current ;
    package CYGPKG_FS_JFFS2 current ;
    package CYGPKG_ERROR current ;
    package CYGPKG_LINUX_COMPAT current ;
    package CYGPKG_IO current ;
    package CYGPKG_CRC current ;
    package CYGPKG_LIBC_STRING current ;
};

cdl_option CYGPKG_IO_FILEIO_DEVFS_SUPPORT {
    user_value 1
};
cdl_component CYGPKG_IO_FLASH_BLOCK_DEVICE {
    user_value 1
};

For example:

    $ ecosconfig new adderII
    $ ecosconfig import jffs2.ecm
    $ ecosconfig tree
    $ make tests

Use with RedBoot

JFFS2 support can be built into RedBoot using the above minimal configuration file. In most cases, the configuration settings will then make all the adjustments necessary.

However note that a build of RedBoot which includes JFFS2 with RedBoot, is likely to require more Flash space for its own image, as well as much more RAM space to run. The latter is particularly important to note given that this can reduce the size of the program image which can be loaded into RAM from a JFFS2 filesystem.

Particularly large JFFS2 filesystems, or filesystems with a large number of nodes, require more RAM to be used for JFFS2's in-memory data structures. As such, the value of the configuration option controlling the size of the RedBoot heap (CYGMEM_REDBOOT_WORKSPACE_HEAP_SIZE) may need to be increased in such cases. JFFS2 will already make the default size of this heap occupy 192KiB of RAM.

Secure Erase

The eCosPro® port of JFFS2 includes a Secure Erase feature. This feature allows the application to ensure that when a file is deleted, its contents are fully erased from the flash.

Usually deleted files persist in Flash for an indeterminate period of time, marked as obsolete. The possible solution taken with other filesystems of trying to overwrite the file data before deletion does not work with JFFS2, as JFFS2 will still retain the old file data in Flash, but writes additional nodes with a higher node version number, and rendering the previous data obsolete. Therefore the Secure Erase functionality can be used to guarantee that a deleted file will have its past contents wiped from the Flash.

Methodology

Because obsolete data for a file could exist in any block, the only way to achieve this is to ensure that every block (other than bad or completely free blocks) is wiped, taking care to relocate live data. This effectively means methodically garbage collecting and erasing every flash block used by the filesystem.

Operation time

For a filesystem which almost completely consists of used flash blocks the secure erase process could take a considerable amount of time during which the filesystem cannot be used for other operations. Therefore it is strongly recommended that the garbage collection thread support is enabled with an appropriate value for CYGNUM_JFFS2_GC_THREAD_TICKS. With the garbage collection thread running, more blocks are likely to be completely clean, or at least partially garbage collected, thus reducing the time for secure erasure.

Usage

Support for the secure erase functionality must first be enabled with a CDL configuration option - CYGOPT_FS_JFFS2_SECERASE.

Then a secure erase operation can be performed on the filesystem with a cyg_fs_setinfo() function call using the FS_INFO_SECURE_ERASE config key. This call can be invoked specifying any file or directory within the filesystem including its mount point, although the operation itself will take place on the entire filesystem.

Example 2. Secure erase usage

#include <cyg/fileio/fileio.h>
#include <errno.h>
#include <stdio.h>
    …
    int err;
    err = cyg_fs_setinfo("/fs", FS_INFO_SECURE_ERASE, NULL, 0);

    if (ENOERR != err)
    {
      printf( "Secure erase failed: %d\n", strerror(err) );
      …
          

Testing JFFS2

JFFS2 comes with a number of tests that may be run as normal eCos test applications. To run these tests, you should create a FIS partition in RedBoot named “jffs2test”.

Without a FIS partition of this name, you must set the CDL configuration options CYGNUM_FS_JFFS2_TEST_OFFSET and CYGNUM_FS_JFFS2_TEST_LENGTH. The tests will attempt to use the region identified by that offset/length combination, but will first check it is blank, and will report a test failure if it is not.

When the tests run, they will erase the Flash test area (usually the “jffs2test” FIS partition) in its entirety, so do not use an existing JFFS2 partition in this space.

The tests are designed to test both general features of JFFS2, as well as do a limited stress-test JFFS2 in the presence of multiple threads.

More specifically, the jffs2-fileio1 test checks a wide variety of file system operations including creating and removing files and directories, scanning directories, and reading and writing file contents. It also repeats to verify that unmounting and remounting works.

The jffs2-fseek1 test verifies file seek operations on JFFS2 files, using standard I/O C library calls.

Test files with names of the form jffs2-NtNf verify operation with varying numbers of threads, and varying numbers of files.

The test jffs2_3 is specifically to verify operation of the garbage collection code, and performs a small set of operations repeatedly to do so. It also gives an opportunity for the garbage collection thread to be tested, if enabled.

The test jffs2-secerase1 is specifically to verify operation of the secure erase facility, if the CYGOPT_FS_JFFS2_SECERASE CDL configuration option has been enabled. It also provides further testing of the garbage collection code and the garbage collection thread.

2017-02-09
Documentation license for this page: eCosPro License