Difference between revisions of "Solution to App Catalog Installation Limit"

From WebOS Internals
Jump to: navigation, search
(Files Confirmed ''Not'' to Work)
Line 613: Line 613:
  
 
== Files Confirmed ''Not'' to Work ==
 
== Files Confirmed ''Not'' to Work ==
 +
:com.palm.app.pdfviewer
  
 
== Discuss ==
 
== Discuss ==

Revision as of 09:15, 9 October 2009

Based on rwhitby's findings that temporarily moving /var/usr/palm/applications will allow installing apps from the App Catalog, here's a permanent way to keep the apps on (much larger) /media/internal by using links. This will permanently save space on /var.

It's unfortunate that Palm has not resolved the issue in the 1.2 update. This is a tactical workaround that could potentially work long term, also saving space on /var, but this needs support by the homebrew community to add in the homebrew installer apps.

Overview

This solution involves creating a (hidden) directory in the /media/internal area, moving selected applications to the newly created directory, and then creating a symbolic link in the /var/usr/palm/applications directory pointing to the new location. This frees up the disk space from the relatively limited /var volume to the larger /media/internal. The included script (which must be created) will move the files and create the appropriate link. It will also provide information on the size of the applications stored in the /var/usr/palm/applications directory.

Solution (symbolic link method)

Create a script to move apps to new home and create link. You can use an editor or use the following cat command to paste the code, for those not familiar with editors.

Here's how to do it for those comfortable with accessing Linux...

- Enter Linux mode on Pre through a computer (linux commands are in bold)

- Set read/write to allow creating script
mount -o remount,rw /

- create a script using cat command, paste the code below and then do 'ctrl c' on the keyboard to break out.
cat >/usr/bin/mvapp

- paste the code below here, press ctrl-c on keyboard to exit after pasting the code.

- when you see a linux prompt, set command to be executable
chmod 755 /usr/bin/mvapp


code:

#!/bin/sh

# This code is open for re-use with no restrictions.  xorg
# This is a working proof of concept script still in development.
# Intent is for someone to port to a webOS app.
# Use at your own risk.

#-------------------------------------------------------------------------#
# versions:
# 0.1.0 - original (xorg)
# 0.1.1 - added unlink and clean functions (daventx)
# 0.1.2 - added bulkmv function, allows moving many apps (xorg)
# 0.1.3 - added option for tar backups (xorg)
# 0.1.4 - added listmoved function to show apps already moved (xorg)
# 0.1.5 - added restoreall function, couple cleanup items (xorg)
# 0.1.6 - fixed to show usage if no appname supplied to link/unlink (xorg)
# 0.1.7 - added cleanexit (w/mount ro /) 
#       - added exit code documentation for javascripts calling this (xorg)
#-------------------------------------------------------------------------#

#-------------------------------------------------------------------------#
# variables: these are globally available to all functions
#-------------------------------------------------------------------------#
COMMAND=$1
APP=$2
MEDIA=/media/internal/.apps
VAR=/var/usr/palm/applications

# Backup keeps file attributes but uses more /media space. Recommend doing backups.
BACKUP=1  # set to 1 for tar backups, 0 to disable
BACKUPDIR=/media/internal/.appbackups

#-------------------------------------------------------------------------#
# exit codes for javascripts:
# function usage:
# 1 - normal usage error
#
# function cleanapp:
# 0 - normal exit
#
# function linkapp:
# 10 - app name not supplied
# 11 - link already exists
# 12 - app does not exit in VAR
# 13 - copy failed from VAR to MEDIA
# 14 - removing app from VAR failed
#
# function unlinkapp:
# 20 - app name not supplied
# 21 - app doesn't exit on MEDIA
# 22 - tar restore failed
# 23 - copy failed
# 24 - remove failed
#
#-------------------------------------------------------------------------#

#-------------------------------------------------------------------------#
# function: cleanexit - exit with cleanup items
#-------------------------------------------------------------------------#

cleanexit () {
 code=$1

# put / back to read only
  mount -o remount,ro /

# Uncomment if you want verbose exit codes
# echo "exit code: $code"

 exit $code
}
# end of cleanexit function


#-------------------------------------------------------------------------#
# function: usage - show command usage options
#-------------------------------------------------------------------------#
usage () {
   exitcode=$1
   if [ ! $exitcode ]
   then
     exitcode=1  # default exit code for usage, otherwise exit with incode
   fi

   echo "Usage: mvapp link domain.appname    - move app to media and link"
   echo "Usage: mvapp unlink domain.appname  - restore app to var, remove link"
   echo "Usage: mvapp clean domain.appname   - remove app dir and links"
   echo "Usage: mvapp list                   - list all apps sorted by size"
   echo "Usage: mvapp bulkmv                 - move/link bulk apps"
   echo "Usage: mvapp listmoved              - list apps that have been moved"
   echo "Usage: mvapp restoreall             - restore all apps to original"

  cleanexit $exitcode
}
# end of usage function


#-------------------------------------------------------------------------#
# function: cleanapp - removes symbolic links and folder in media and var
#-------------------------------------------------------------------------#

cleanapp () {

 mount -o remount,rw /

 # exit to usage if no app name supplied
 if [ ! $APP ]
 then
   usage
 fi

 # Optional
 echo "This will remove $APP from both $VAR and $MEDIA."
 echo "You should first attempt to remove the app using the official Pre methods."
 echo "Are you sure you want to remove $APP? [y/N]: "
 read answer
 case $answer in
   [Yy]*) continue;;
       *) cleanexit 0;;
 esac

 if [ -h $VAR/$APP ]
 then
   echo "Removing all traces of $APP."
 else
   echo "$APP does not exist..."
   usage 10
 fi

 echo "Size of $VAR before cleanup... "
 du -sh $VAR

 if [ -d $MEDIA/$APP ]
 then
     rm -r $MEDIA/$APP
      echo "Removed directory" $MEDIA/$APP
 fi
 if [ -d $VAR/$APP ]
 then
      rm -r $VAR/$APP
      echo "Removed directory" $VAR/$APP
 fi
 if [ -L $VAR/$APP ]
 then
      rm -r $VAR/$APP
      echo "Removed link" $VAR/$APP
 fi
 if [ -f $BACKUPDIR/$APP.tgz ]
 then
      rm -r $BACKUPDIR/$APP.tgz
      echo "Removed tar backup" $BACKUPDIR/$APP.tgz
 fi


 # rescan luna in case it's needed
 luna-send -n 1 palm://com.palm.applicationManager/rescan {} >/dev/null 2>&1
 echo "$APP directories and links removed."
 echo "Size of $VAR after cleanup... "
 du -sh $VAR
 cleanexit 0
}
# end of cleanup function


#-------------------------------------------------------------------------#
# function: listapps - list the size of each app, sort showing largest last
#-------------------------------------------------------------------------#
listapps () {
 du -s $VAR/* | sort -n
 cleanexit 0
}
# end of listapps function

#-------------------------------------------------------------------------#
# function: listmoved - list apps moved/linked, sort showing largest last
#-------------------------------------------------------------------------#
listmoved () {
 du -sk $MEDIA/* | sort -n
 cleanexit 0
}
# end of listmoved function


#-------------------------------------------------------------------------#
# function: linkapp - move the app to media and create symbolic link
#-------------------------------------------------------------------------#
linkapp () {

 if [ ! $APP ]
 then
      echo "No application supplied..."
      usage 10
 fi

 if [ ! -d $MEDIA ]
 then
      mkdir $MEDIA
 fi

 if [ -h $VAR/$APP ]
 then
      echo "Link already exists for... ${APP}"
      cleanexit 11
 fi

 mount -o remount,rw /

 if [ -d $VAR/$APP ]
 then
   echo "Moving $APP to $MEDIA..."
 else
   echo "$APP does not exist..."
   usage 12
 fi

 # Backup using tar if enabled
 if [ $BACKUP ]
 then
   if [ ! -d $BACKUPDIR ]
   then
    mkdir $BACKUPDIR
   fi
   echo "Backing up $APP to $BACKUPDIR..."
   tar czf $BACKUPDIR/${APP}.tgz $VAR/$APP
 fi

 mount -o remount,rw /

 echo "Size of $VAR before move... "
 du -sh $VAR

 # move over to USB drive
 cp -rp  $VAR/$APP $MEDIA
 if [ $? != 0 ]
 then
  echo "Copy failed. Leaving app in $VAR."
  rm -r $MEDIA/$APP
  cleanexit 13
 fi

 rm -r $VAR/$APP
 if [ $? != 0 ]
 then
  echo "Remove failed. Leaving app in $VAR."
  rm -r $MEDIA/$APP
  cleanexit 14
 fi

 # create the symbolic link
 ln -s $MEDIA/$APP $VAR/$APP

 # rescan luna in case it's needed
 luna-send -n 1 palm://com.palm.applicationManager/rescan {} >/dev/null 2>&1

 echo "$APP moved and linked."
 echo "Size of $VAR after move... "
 du -sh $VAR
}
# end of linkapp function


#-------------------------------------------------------------------------#
# function: unlinkapp -  restore the app to var and remove symbolic link
#-------------------------------------------------------------------------#
unlinkapp () {

 if [ ! $APP ]
 then
   echo "No application supplied..."
   usage 20
 fi

 mount -o remount,rw /

 if [ -d $MEDIA/$APP ]
 then
   echo "Restoring $APP..."
 else
   echo "$APP does not exist..."
   usage 21
 fi

 echo "Size of $VAR before move... "
 du -sh $VAR

 # remove the old symbolic link
 rm -r $VAR/$APP

 # move to original location or restore from tar if it exists
 if [ -f $BACKUPDIR/$APP.tgz ]
 then
  cd /
  tar xzf $BACKUPDIR/$APP.tgz
  if [ $? != 0 ]
 then
   echo "Tar restore failed. Remove and restore app using official webOS/Pre methods."
   cleanexit 22
  else
   rm -r $BACKUPDIR/$APP.tgz
  fi
 else
  cp -r  $MEDIA/$APP $VAR
  if [ $? != 0 ]
  then
   echo "Copy failed. Leaving app in $MEDIA."
   cleanexit 23
  fi
 fi

 rm -r $MEDIA/$APP
 if [ $? != 0 ]
 then
  echo "Remove failed. Leaving app in $MEDIA."
  rm -r $VAR/$APP
  cleanexit 24
 fi

 # rescan luna in case it's needed
 luna-send -n 1 palm://com.palm.applicationManager/rescan {} >/dev/null 2>&1

 echo "$APP moved and unlinked."
 echo "Size of $VAR after move... "
 du -sh $VAR
}
# end of unlinkapp function

#-------------------------------------------------------------------------#
# function: bulkvm -  move/link many apps
#-------------------------------------------------------------------------#
bulkmv() {
 echo
 echo
 echo "This allows moving many apps, asking which you'd like to move."
 echo "Starting with the largest apps."
 echo


 mount -o remount,rw /
 cd $VAR

 for i in `du -s * | sort -nr |cut -f 2`
 do
    export APP=$i
    SIZE=`du -sh $APP |cut -f 1`
    echo "Size of $APP is $SIZE."
    echo "Would you like to move and link $APP ? [y/N/q]: "
    read answer
    case $answer in
    [Yy]*) linkapp;;
    [Qq]*) cleanexit 0;;
        *) echo "$APP not moved."
           continue;;
    esac

    echo
 done
}
# end of bulkmv function

#-------------------------------------------------------------------------#
# function: restoreall - restore all apps, back to /var
#-------------------------------------------------------------------------#
restoreall() {
  echo "This will restore all applications back to original location"
  echo "and remove the links.  Are you sure you want to continue? [y/N]:"
  read answer
  case $answer in
    [Yy]*) continue;;
        *) cleanexit 0;;
  esac

  ls $MEDIA | while read APP
  do
    echo "Restoring $APP and unlinking..."
    unlinkapp
  done
}
# end of restoreall function


#-------------------------------------------------------------------------#
# main - begins here
#-------------------------------------------------------------------------#

case $COMMAND in
"clean")
   cleanapp
   ;;
"list")
   listapps
   ;;
"listmoved")
   listmoved
   ;;
"link")
   linkapp
   ;;
"unlink")
   unlinkapp
   ;;
"bulkmv")
   bulkmv
   ;;
"restoreall")
   restoreall
   ;;
*)
   usage 0
   ;;
esac

cleanexit 0



- Be sure to set execute permissions if you missed the step above... chmod 755 /usr/bin/mvapp

Precautions

So far, no one has reported an app that has issues due to linking. These are best practices to avoid issues.

- Be selective about what you move. You may not want to move apps that store important information to you such as password lockers, memo apps, EverNote, Agenda, etc. Games, web content apps and information viewing apps should be safer to move as there is no data at risk.

- Close apps before moving them. Do not move an app while it is open and running. I actually tested this and figured some apps would break but none have yet. It is still not wise to move apps while they are open and running.

Commands

To find the largest apps in /var/usr/palm/applications

mvapp list

It shows the largest apps last... IE....
8352 com.apnews.webos
8512 com.fandango.app.fandango
8672 com.palm.app.musicplayerremix
10304 com.shortcovers.palm.pre
10432 com.fusioncreativestudios.deadman
10656 com.ulocate.app.where


To move and link an app to /media

mvapp link domain.appname

Example: mvapp link com.ulocate.app.where

The app should now work in the new location thanks to the link. Test each app to make sure it works before doing another. Looping the script to do every app in the dir or to link the entire app dir is not recommended. File attributes are lost on the move, which some apps may need. You need to test each app after a move.

Continue moving apps until it reports that /var.../applications is about 25MB or less.

There is an option to turn on/off tar-based backups. See the BACKUP variable in the script. If set to 1, it will perform a tar backup. This takes up more space on /media but if you need to restore, it puts it back with full file attributes. Recommend keeping this on.


To move and link many apps

mvapp bulkmv

This will show largest app first, ask if you want to move/link and then moves on to the next largest app. If you answer no to an app, it will skip to the next. Answer 'q' to quit. Is easiest to use this method directly on a device using the Terminal app.


To revert the move and delete the link

mvapp unlink domain.appname

The application will be moved back to the original directory. File attributes are lost on the move if tar backup is turned off, which some apps may need. You need to test each app after a move. If tar backup is turned on (default - see BACKUP variable in script), file attributes will be restored.


To list apps that have been moved and linked...

mvapp listmoved

This shows a list of apps that have been moved and linked to /media.


To restore all moved apps back to original location

mvapp restoreall

If backup was turned on, this will restore ALL applications that have been moved/linked, back to the original location in /var with original attributes (using tar). If backup was not turned on, it will still attempt to move the applications back to /var using the files in /media, but without original file attributes. This may take several minutes to complete. It will show each application progress.


To cleanup and remove directories and symlinks

mvapp clean domain.appname

If you have any issues, first attempt to remove the app from the App Catalog or Homebrew app installer. Then issue the 'clean' command to remove the applications directories at both locations and also the symlink. You can then reinstall the app from the App Catalog or any Homebrew App installer.

Contingency Plan

If you have problems with an application, follow these steps...
-- Close the application if open
-- mvapp unlink domain.appname
-- Try using the App

If the application still does now work correctly...
-- Remove the App using official methods
-- mvapp clean domain.appname
-- Reinstall through the App Catalog or Homebrew installer app

Proposal to Dev Community

This is a formal proposal to the Dev Community suggesting that PreWare, WebOS Quick Install and other Pre installer apps provide an option to move any app in /var to the /media fs and create a link similar to the code above.

The Homebrew Community somewhat created part of the storage problem so needs to come up with their own solution. The symbolic link proponents propose that Homebrew apps be moved with a link to /media/internal by default and physically use /var only if needed (per conditions stated below). The developer would put a flag in the package (or some other method during submission) to state their app is able to run linked to /media or if it specifically needs to physically be on /var. Will propose additions to Packing Standards to support /media links. The homebrew installer apps could then automatically do the move/link if the package is flagged for it.

Candidate apps for moving to /media
- apps that do not depend on file attributes (such as executable bit)
- apps that do not perform data operations to home app directory when device is USB mounted

Exceptions for maintaining apps on /var
- apps that depend on file attributes
- apps that won't work well when device is USB mounted, such as performing data or DB operations in home app directory

(Please update with other known candidates/exceptions)

Benefits over other methods

Resizing /var
One challenge with resizing /var is that it will still have a fixed static limit - how do you decide how much to increase it? Many will still probably hit the limit or waste space if setting too high. There is also the warning from Palm that resizing var may interfere with future updates. The link method allows to dynamically use the /media partition, so there is no need to set a specific size dedicated to apps. If the USB drive is filled, users can decide if they use the space for media or apps on the fly.

AppPath in /etc/luna.conf
I proposed a while back adding an AppPath to luna.conf to include apps stored on /media. Some apps would not work because some apparently reference /var. IE, the vampire/mafia/quest series could not access graphics. The symbolic link fixed this because apps think they are in /var.

Link/mount all of /var/usr/palm/applications/ to /media/internal
If some apps rely on file attributes this won't work since file attributes are lost when moved to FAT fs. While this is rare, it probably isn't wise to force all /var apps to /media. Selectively moving apps one at a time is less risky. Update: Have found that "PDF View" app does not allow to be linked, so it appears that selectively moving apps is necessary.

Creating a loopback filesystem to a virtual file located on /media/internal
This has been worked on here and still has potential. Unfortunately it locks out USB mount and media sync altogether. If a workaround can be found with low risk, this may be the most ideal solution.

Risks, Issues, Dependencies

- File attributes of linux fs are not copied over to fat fs. May not work with apps that rely on file attributes. In general, javascript apps will likely work fine located on /media. No one has reported an app (yet) that does not work.

- Some apps may not behave well if USB drive is mounted to computer, though I've tested several that behaved fine. Linux type background services probably would not work well so probably should not be moved. Apps that do IO to the home directory of the app while USB mounted may have issues when located on /media.

- Jason Robitaille has indicated that a future version of WebOS Quick Install will include a routine to resize the /var partition. He has also indicated that doing so will wipe the /media/internal area, so any applications stored there would need to be backed up first. It might be good to be able to reverse the process of this solution. This could be done either by adding to this mvapp script, or creating a new one.

Future Versions

Plans for future versions...

- The script will soon be modified to be friendly for javascript calls... no prompting and no output, just result codes. Will still maintain user interaction capability as well.

Files Confirmed to Work

com.accuweather.palm
com.asc.funfactsw
com.biocandy.prejeweled
com.bitsofgodsoftware.simplebible
com.bytesequencing.pre.balls
com.bytesequencing.pre.spades
com.cakefight.sudoku
com.deliciousmorsel.twee
com.engineequalscar.games.mines
com.fandango.app.fandango
com.flixster.app.movies
com.friendsflow.friendsflowbeta
com.fusioncreativestudios.deadman
com.jackintheboxsoftwareinc.app.mobios
com.landgrabgames.zilch.lite
com.longbeachit.gototoollite
com.mediafly.mediafly-pre
com.mrjspeed.toppleball
com.myappcatalog.cardmessagesbeta
com.myappcatalog.guitaristsreference
com.myappcatalog.superhangmanbeta
com.orbsix.app.magic8ball
com.palm.app.agenda
com.palm.pandora
com.palm.repomanager
com.quigley-jones.preopoly
com.sachersoft.wordwhirllite
com.splashdata.app.mcraig
com.stickywhipit.yatz
com.tj.solitaire
com.wikihow.wikihowapp
net.keenstudios.app.checkers
ws.junk.blocked
ws.junk.dotgame

Files Confirmed Not to Work

com.palm.app.pdfviewer

Discuss

Discuss in the Discussion tab or PreCentral...

http://forums.precentral.net/web-os-development/205649-resolution-app-catalog-install-limit-proposal.html

Your Experiences

Please post your experiences, good or bad. I'd like to get any kinks worked out before attempting to turn this into a webOS app.

- How many apps did you move?
- Did you find an app that won't work linked?
- Did you move any back? (please test)
- How far down did you have to get /var.../applications down in MB before you could start adding apps from the App Catalog?

Credits

xorg - initially developed script and proposal. maintainer of this page.
daventx - added unlink and clean functions.