Topic: Maximum Filesystem Size?

I have just upgraded my home file server from a 1.7 TB filesystem to on the new server 5.3 TB. So I have swapped my NFS mount to it on chumby.

Via SSH the chumby sees everything fine on df and ls e.g:

Chumby:/psp # df -h
192.168.1.201:/home/multi
                          5.3T      1.5T      3.9T  27% /mnt/nfs

But using the Music -> My Music Files and browse to /mnt/nfs just shows an empty directory. So I strace'd the control panel. It shows:

4462  open("/mnt/nfs", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 19                                 
4462  fstat64(19, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0                                                   
4462  getdents(19, /* 2 entries */, 32768) = 32                                                                   
4462  getdents(19, 0x253f20, 32768)     = -1 EOVERFLOW (Value too large for defined data type)                     
4462  close(19)                         = 0                                                         

Now I probably need to set options on strace to see what the fstat64 is actually calling to get this  EOVERFLOW from getdents. But give the source for all this is unavailable, there isn't probably much I can do about it, I guess it's maybe flash player itself that is the issue.

But just wondered if anyone has seen this before or knows if this can be worked around at all?

Re: Maximum Filesystem Size?

The source for everything but the Flash Player is available at http://files.chumby.com/source

The Flash Player is linked against standard libraries - from what I can tell, glibc had a bug that would incorrectly return an overflow of the getdents() call under certain condition, and we might be using that. It might also be the case that the device simply does not have enough available memory to maintain the file directory for a volume of that size.

Re: Maximum Filesystem Size?

Doing a little more research here, this does look like some sort of size limit due to the chumby's use of a 32-bit Linux (naturally, since the processor is 32-bit) and a file system which may exceed those limits - files >2GB, for instance.

Re: Maximum Filesystem Size?

I have done a bit more research.

It's not a large file problem, as the files are identical as to when they were on original smaller fileserver. This top level directory also just contains 25 items. So it must be an issue with a much larger filesystem.

Now I had a bit more of a look around and it seems to be to do with XFS (this is the default on Centos 7) and 64 bit inodes.

This is a good article:
http://www.tcm.phy.cam.ac.uk/sw/inodes64.html

To quote:

Modern filesystems are starting to use 64 bit inodes, rather than 32 bit ones. Recent versions of XFS do so for filesystems of more than 1TB, and other large filesystems such as Lustre use them too. NFS fully supports them too, so one can find them when mounting remote filesystems.

And:

Two very common system calls, stat() and readdir return an inode number. If the 32 bit version of these calls is used on a filesystem with 64 bit inodes, then the call may fail with EOVERFLOW.

I write "may fail" rather than "will fail" as the failure will happen only if the particular inode number cannot be expressed in 32 bits.

The "solution" from the article that may work is:

Ask the kernel to lie

Boot with the option nfs.enable_ino64=0 (assuming the 64 bit inodes are coming from an NFS mounted filesystem). This causes the kernel to return non-unique 32 bit inode numbers to all applications, 32 bit or 64 bit, for all NFS mounted filesystems. It can be a useful test that a problem is caused by 64 bit inode numbers, but is hard to recommend as some 64 bit applications might be confused by the lie (which is to xor the top 32 bits of the inode number with the bottom 32 bits).

The caveat shouldn't be an issue as Chumby is all 32 bit.

But they question is does the Chumby kernel support this "nfs.enable_ino64=0" flag? And how do you change the kernel boot flags on a Chumby ?

Re: Maximum Filesystem Size?

I think you need to rebuild the boot loader in order to change the kernel options.

Re: Maximum Filesystem Size?

I tried his user space option on the web link I gave. Unfortunately that doesn't work for me, maybe I have made a mistake in porting it to Chumby (data structures) but does seem to work with programs just doesn't fix the issue. It maybe the failing call to getdents isn't called through readdir() that this shim is (hopefully) trapping.

So I thought I'd give the kernel flag a look. So here the questions start?

1/ I have a Chumby One, so I downloaded the source code for bootstream-1.0.tgz  , Falconwing Software version 1.0.4, build 1.0.2913, as the build version I have from /etc/firmware_build is 1.0.3454, but that doesn't list bootstream source for this version. So I presume the previous version that has bootstream 1.0.2913  was used in this version too?

2/ In the source code for  bootstream it says:

chumby_stub/include/linux.h:#define DEFAULT_LINUX_CMDLINE "console=ttyAM0,115200 init=/linuxrc root=/dev/mmcblk0p2 rootfstype=ext3 ro rootwait lcd_panel=lms350 ssp1=mmc line=1 sysrq_always_enabled"

This is called chumby_boot/src/shell_funcs.c:

char *cmdline       = DEFAULT_LINUX_CMDLINE;

But then says

 if(argc > 2)
        cmdline = argv[2];

And given the default doesn't match the contents of /proc/cmdline

console=ttyAM0,115200 init=/linuxrc root=/dev/mmcblk0p3 rootfstype=ext3 ro rootwait chumbyrev=06 ssp1=mmc sysrq_always_enabled logo.brand=chumby

There is another way of passing the kernel command line?

3/ Even if I need to compile a new boot loader I tried to do this and get:

Makefile:4: ../config/config.mk: No such file or directory

Where does this come from in the source codes?

4/ Is there any documentation on building the bootloader and how to change it?

Re: Maximum Filesystem Size?

Actually hold on all the bootloader stuff I may have just got the user space shim working.

Not sure if I can post the whole of the inode64.c here, it's quite long?

I'd post the binary too if there is a built in way to do this in the forum?

Re: Maximum Filesystem Size?

I think I have managed to get this working. Below is the source that worked for me. I have used primitive types in the stat structures, as I couldn't get the original way working on Chumby. I compiled this on the Chumby itself. I tried putting the software shim in just for the flash player but still wouldn't play my mp3's. There must be other processes involved in playback that this shim wasn't catching. There's maybe a better place than /etc/init.d/rcS.background, but this assured  I'd catch any processes that need this. Again no guarantees, I have only briefly tested this.

The bulk of the work was of course the original author of this MJ Rutter.
http://www.tcm.phy.cam.ac.uk/sw/inodes64.html

I thought I'd post just in case anyone else brings up a RHEL/Centos 7 NFS server with XFS or any newer filesystem on any distro with 64 bit inodes that the Chumby NFS client struggles with (well it's the 32 bit apps to be honest).

Instructions in the comments section are what I did for Chumby, altered from the original instructions.

 

/* A library for fixing programs which object to 64 bit inodes on Linux.
 *
 * (c) 15/8/2014, MJ Rutter
 * Altered for Chumby by C Simpson 23/08/2015
 *
 * Can be distributed under the Gnu Public Licence version 2
 * ( http://www.gnu.org/licenses/gpl-2.0.html )
 *
 * WARNING!! USE AT OWN RISK -- AUTHOR ACCEPTS NO RESPONSIBILITY
 * IF THIS DESTROYS YOUR DATA, OR CAUSES ANY OTHER FORM OF INCONVENIENCE
 * OR DISASTER. AUTHOR RECOMMENDS YOU DO NOT USE IT.
 * 
 *
 * Problem: if a filesystem has 64 bit inodes, 32 bit programs calling
 *  *stat() or readdir() will get EOVERFLOW unless they have been compiled
 *  with large file support. However, it is likely that the only field which
 *  overflows is the inode number, which is usually ignored.
 *
 * This library forces the inode number to 32 bits, and returns the rest
 *   of the structure correctly. It is similar to mounting an NFS filesystem
 *   with the kernel parameter nfs.enable_ino64=0
 *
 * To build:
 *
 * 1/ Create a linker script called "vers" containing:
 *
GLIBC_2.4 {
  global:
    readdir;
    __fxstat;
    __xstat;
    __lxstat;
};
 *
 * 2/ compile: 
 *
 *  gcc -c -fPIC inode64.c
 *
 * 3/ link
 *
 * ld -shared --version-script vers -o inode64.so inode64.o
 *
 * 4/ Copy inode64.so to /root. Insert  export LD_PRELOAD  line below into /etc/init.d/rcS.background
 
* e.g. 
 udevadm settle
 
 
 export LD_PRELOAD=${LD_PRELOAD:+${LD_PRELOAD}:}/root/inode64.so
 ##########################################################
 #####
 #####
 ##### Greetings!  If you're coming up with your own userhook0
 
 
 *
 *
 * Bugs:
 *
 * 1/ No attempt is made to fix readdir_r()
 *
 * 2/ readdir() overwrites its buffer on every call, even on calls to
 *      different directory streams
 *
 * If compiled with INPLACE defined (the default), then the returned structure
 *   from readdir64() is modified in place, and this bug does not apply. Does
 *   POSIX permit modifying the returned structure? Unclear to me...
 *
 */

#define INPLACE

#include<stdlib.h>
#include<bits/types.h>
#include<stdint.h>
#include<unistd.h>
#include<errno.h>

struct dirent32 {
  uint32_t       d_ino;       /* inode number */
  uint32_t       d_off;       /* not an offset; see NOTES */
  unsigned short d_reclen;    /* length of this record */
  unsigned char  d_type;      /* type of file; not supported
                                 by all file system types */
  char           d_name[256]; /* filename */
};

#ifdef INPLACE
struct dirent64 {
  uint64_t       d_ino;       /* inode number */
  union {
    uint64_t       d_off;       /* not an offset; see NOTES */
    uint32_t       i32[2];
  } fudge;
  unsigned short d_reclen;    /* length of this record */
  unsigned char  d_type;      /* type of file; not supported
                                 by all file system types */
  char           d_name[256]; /* filename */
};

#else
struct dirent64 {
  uint64_t       d_ino;       /* inode number */
  uint64_t       d_off;       /* not an offset; see NOTES */
  unsigned short d_reclen;    /* length of this record */
  unsigned char  d_type;      /* type of file; not supported
                                 by all file system types */
  char           d_name[256]; /* filename */
};
#endif

struct stat32 {
        unsigned long long  st_dev;
        unsigned short   __pad1;
        unsigned long  st_ino;
        unsigned int st_mode;
        unsigned int st_nlink;
        unsigned long st_uid;
        unsigned long st_gid;
        unsigned long long st_rdev;
        unsigned short pad2;
        unsigned long st_size;
        unsigned long  st_blksize;
        unsigned long st_blocks;
        unsigned long  st_atim;
        unsigned long  st_atim_nsec;
        unsigned long  st_mtim;
        unsigned long  st_mtim_nsec;
        unsigned long  st_ctim;
        unsigned long  st_ctim_nsec;
        unsigned long unused1;
        unsigned long unused2;
};

struct stat64 {
        unsigned long long        st_dev;
        unsigned char   __pad0[4];
        unsigned long        __st_ino;
        unsigned int        st_mode;
        unsigned int        st_nlink;
        unsigned long        st_uid;
        unsigned long        st_gid;
        unsigned long long        st_rdev;
        unsigned char   __pad3[4];
        long long        st_size;
        unsigned long        st_blksize;
        unsigned long long st_blocks;        /* Number 512-byte blocks allocated. */
        unsigned long        st_atim;
        unsigned long        st_atim_nsec;
        unsigned long        st_mtim;
        unsigned long        st_mtim_nsec;
        unsigned long        st_ctim;
        unsigned long        st_ctim_nsec;
        unsigned long long        st_ino;
};



typedef struct __dirstream DIR;
struct dirent64 *readdir64(DIR *dirp);

#ifdef INPLACE
struct dirent32 *readdir(DIR *dirp){
  struct dirent64 *ptr;
  int inode,ioff;

  ptr=readdir64(dirp);

  if (!ptr) return NULL;

  inode=ptr->d_ino^(ptr->d_ino>>32);
  ioff=ptr->fudge.d_off;
  ptr->fudge.i32[0]=inode;
  ptr->fudge.i32[1]=ioff;
  ptr->d_reclen-=8;

  return (struct dirent32 *)&(ptr->fudge);
}
#else
struct dirent32 *readdir(DIR *dirp){
  static struct dirent32 d32;
  struct dirent64 *ptr;
  int i;

  ptr=readdir64(dirp);

  if (!ptr) return NULL;

  d32.d_ino=ptr->d_ino^(ptr->d_ino>>32);
  d32.d_off=ptr->d_off;
  d32.d_reclen=ptr->d_reclen-8;
  d32.d_type=ptr->d_type;
  for(i=0;i<256;i++) d32.d_name[i]=ptr->d_name[i];

  return &d32;
}
#endif

int __fxstat64 (int ver, int fd, struct stat64 *buf);

int __fxstat (int ver, int fd, struct stat32 *buf){
  struct stat64 s64;
  int i;

  i=__fxstat64(ver,fd,&s64);
  if (i) return i;

  buf->st_dev=s64.st_dev;
  buf->st_ino=s64.st_ino^(s64.st_ino>>32);
  buf->st_mode=s64.st_mode;
  buf->st_nlink=s64.st_nlink;
  buf->st_uid=s64.st_uid;
  buf->st_gid=s64.st_gid;
  buf->st_rdev=s64.st_rdev;
  if (s64.st_size>>32) {errno=EOVERFLOW; return -1;} 
  buf->st_size=s64.st_size;
  buf->st_blksize=s64.st_blksize;
  buf->st_blocks=s64.st_blocks;
  buf->st_atim=s64.st_atim;
  buf->st_mtim=s64.st_mtim;
  buf->st_ctim=s64.st_ctim;
  buf->st_atim_nsec=s64.st_atim_nsec;
  buf->st_mtim_nsec=s64.st_mtim_nsec;
  buf->st_ctim_nsec=s64.st_ctim_nsec;

  return 0;
}

int __xstat64 (int ver, const char *path, struct stat64 *buf);

 int __xstat (int ver, const char* path, struct stat32 *buf){
  struct stat64 s64;
  int i;

  i=__xstat64(ver,path,&s64);
  if (i) return i;

  buf->st_dev=s64.st_dev;
  buf->st_ino=s64.st_ino^(s64.st_ino>>32);
  buf->st_mode=s64.st_mode;
  buf->st_nlink=s64.st_nlink;
  buf->st_uid=s64.st_uid;
  buf->st_gid=s64.st_gid;
  buf->st_rdev=s64.st_rdev;
  if (s64.st_size>>32) {errno=EOVERFLOW; return -1;} 
  buf->st_size=s64.st_size;
  buf->st_blksize=s64.st_blksize;
  buf->st_blocks=s64.st_blocks;
  buf->st_atim=s64.st_atim;
  buf->st_mtim=s64.st_mtim;
  buf->st_ctim=s64.st_ctim;
  buf->st_atim_nsec=s64.st_atim_nsec;
  buf->st_mtim_nsec=s64.st_mtim_nsec;
  buf->st_ctim_nsec=s64.st_ctim_nsec;

  return 0;
}

int __lxstat64 (int ver, const char *path, struct stat64 *buf);

int __lxstat (int ver, const char* path, struct stat32 *buf){
  struct stat64 s64;
  int i;

  i=__lxstat64(ver,path,&s64);
  if (i) return i;

  buf->st_dev=s64.st_dev;
  buf->st_ino=s64.st_ino^(s64.st_ino>>32);
  buf->st_mode=s64.st_mode;
  buf->st_nlink=s64.st_nlink;
  buf->st_uid=s64.st_uid;
  buf->st_gid=s64.st_gid;
  buf->st_rdev=s64.st_rdev;
  if (s64.st_size>>32) {errno=EOVERFLOW; return -1;} 
  buf->st_size=s64.st_size;
  buf->st_blksize=s64.st_blksize;
  buf->st_blocks=s64.st_blocks;
  buf->st_atim=s64.st_atim;
  buf->st_mtim=s64.st_mtim;
  buf->st_ctim=s64.st_ctim;
  buf->st_atim_nsec=s64.st_atim_nsec;
  buf->st_mtim_nsec=s64.st_mtim_nsec;
  buf->st_ctim_nsec=s64.st_ctim_nsec;

  return 0;
}

Re: Maximum Filesystem Size?

The Flash Player isn't what actually plays the music - it communicates through a pipe with a daemon called "btplayd" to do that. You might need to let both programs use the shim, FP to display the files, and btplayd to play them.

Re: Maximum Filesystem Size?

Yeah I noticed when I strace'd the Flash Player that it was communicating to btplayd. I couldn't see where btplayd was launched from, so I took the crude approach of applying the shim to an rc file so all software would use it. Also I didn't know if any other software (apart from the flashplayer and btplayd) is needed or might be needed to access/play files on the NFS mount.