WordPress Filesystem Abstraction FAQ

I shouldn’t have to be writing this, But as some people will know, I’m utterly sick of repeat questions.

This is a FAQ on the WordPress FileSystem abstraction which i wrote as part of the Summer of Code project to implement a Plugin Upgrader for WordPress (note: the actual upgrader in core is different from what i originally wrote, however ideas and code were lifted and put in their place by a much better coder than I), The content of this post is based on the situation as of today, 20/Feb/09, I fully expect these answers will not remain current in the years to come, But hope they provide some background information on them.

The FAQ:

What methods/transports does WordPress support for accessing the filesystem?

  1. Direct – This method uses direct file IO operations from PHP
  2. FTP – This method connects to your FTP server using the credentials supplied, and performs all IO operations that way
  3. SSH – This method uses the PHP SSH2 extension to connect to the server and manipulate the files there.

What requirements must be met for a transport to be used?

  1. Direct
    • The files which PHP creates MUST be owned by the same user as the WordPress files which have been uploaded
  2. FTP
    • The user will need to supply their FTP credentials in order for it to operate
    • The FTP code branch requires either:
      1. The PHP FTP Extension to be loaded, OR
      2. The FTP Sockets Extension to be loaded, OR
      3. The PHP Function FSockopen() to be available
    • In the case of  the Sockets or Fsockopen case, The 3rd party FTP Class “PemFTP” is used.
    • I should also mention, That this supports Secure FTP as well.
  3. SSH2
    • This was added in WordPress 2.7
    • It requires the PHP SSH2 Extension to be available, This generally requires compiling from source unless you’re lucky enough for it to be available in Pecl for your distribution.
    • Unless you’re running a Dedicated or VPS server, This is not usually available to you due to the above.

Why does my WordPress Install require my FTP Credentials? My other blog doesn’t!!

As above, The Direct Method(Which requires no credentials) only works in a very certain setup, Generally, This is only servers running with suPHP or suExec with PHP as a CGI – Servers running this exist, however, Due to hardware constraints (running these uses extra resources) most hosts which oversell do not use this method.

What can i do?

Move to a different Hoster, Or if thats not an option, Simply entering your FTP details.  You can see a item later in this FAQ for how to hard-code these settings if you want.

No, Seriously, Why cant i use the Direct method? I think i should be able to!!

Because your server configuration is not supported.

Still not taking no for an answer? Ok, Heres the problems with each of the requests people ask for:

  1. I dont want to give my FTP details, I’d rather just chmod 777 everything
    • The problem with this, Is its insecure, WordPress shouldn’t be encouraging non-secure installation methods, Yes, I realise you need to chmod your uploads directory in a few cases, And thats OK, Its a convenience,  Its annoying to have to enter credentials every time you want to upload a file..
    • Yes, WordPress could write to your files in that case, But if you read on to the next point:
  2. I should be able to use it! I’m in the group which PHP runs as! (usually the same as your web server)
    • Thats all nice and well, And that means that PHP can write to your WordPress folder, However, Its got one big gotcha, Whilst it can write to the directory, Theres an issue: The files PHP create will be owned by the webservers username, And the group will be set to that group, the files will NOT be owned by your username.
    • So what i hear you say, I dont care if its not owned by my username, Ok, 2 issues at play here now:
      • You will encounter errors upon attempting to delete the files via FTP, Simply due to the fact you do not own the files.
      • Much on the same path, Because you do not own the files, If you have a Disk quota on the webserver, some encarnations will miss your files and not count it (Hooray! I hear you yell, Thats all good and fine, But if you dont like paying for disk space, find somewhere else that will sell you the even more oversold disk space!)

My FTP Doesnt work!

Ok, This isnt too common, But sometimes PHP’ will have dodgy extensions, or configurations, infact, Its VERY common for a hoster to have a crap install of PHP, the FTP extension and CURL are common culprits for being badly configured, Unfortunately, Theres not much you can do to fix stupid hosts, other than complain.

As a work around, You can use a plugin, or a filter yourself, to work around such circumstances, While I never intended this plugin for end-users, If enough people require it, I’ll separate the needed functionality into a separate Plugin:

Introducing Core Control, Core control is a plugin which gives some extra control over certain items in WordPress, Mainly Filesystem access and HTTP access. Utilising this plugin, You can disable certain Filesystem access methods, Simply install it (The Plugin installer works great for this, Try it again if you’ve disliked the search in the past, The new Search system that went live on WordPress.org recently has improved things greatly there) and enable the Filesystem Module, Once thats done, You can simply disable the current primary transport (Hopefully its the Direct or FTP Extension method) and see if the next in line works for you:

Core Control: Disabled FTP Extension
Core Control: Disabled FTP Extension

I run FTP/SSH on a non-default port!, How can i use this with WordPress?

You can include the port in the hostname, For example, Instead of writing “dd32.id.au” i could use “dd32.id.au:4567” if i was to run on port 4567

FTP Host on alternate Port
FTP Host on alternate Port

WordPress complains that it cant find my WordPress installation!

Whilst this should be highly uncommon (I Hope!), It can happen sometimes, So read on to the next item for a solution.

I have to use the FTP/SSH method, But i dont want to enter my password every time

WordPress also supports the use of defining a selection of constants to ease the sitatuation for a few select circumstances, The supported constants (and a description of each) are:

Constant: Description: For Example:
FTP_HOST The hostname of the server to connect to define(‘FTP_HOST’, ‘dd32.id.au’); or

define(‘FTP_HOST’, ‘dd32.id.au:4567’);

FTP_USER The username to connect with define(‘FTP_USER’, ‘dd32’);
FTP_PASS The password to connect with, WordPress will remember all your other settings by default, However, It will forget your password every time you provide it (ie. it is not stored on the server), You may use this to hard-code your FTP password define(‘FTP_PASS’, ‘*************’);
FTP_PUBKEY (SSH2 only) The path to the Public Key to use for the connection define(‘FTP_PUBKEY’, ‘/var/key.pub’);
FTP_PRIKEY (SSH2 only) The path to the Private Key to use for the connection define(‘FTP_PRIKEY’, ‘/var/key’);
FTP_BASE The path on the FTP server to the WordPress files, This is absolute path in the FTP session, NOT an absolute path on the server. This should be set to the ABSPATH folder define(‘FTP_BASE’, ‘/public_html/wordpress/’);
FTP_CONTENT_DIR The path to the Content directory on the server, This is mainly useful for times when WP_CONTENT_DIR has been set, However, it should work without it in most cases. The warnings in the previous item hold true. define(‘FTP_CONTENT_DIR’, ‘/public_html/wordpress/wp-content/’);
FTP_PLUGIN_DIR The path to the Plugins directory on the server, This is mainly useful for times when WP_PLUGINS_DIR has been set, However, It should workout this in most cases. The warning for the previous items hold true. define(‘FTP_CONTENT_DIR’, ‘/public_html/wordpress/wp-content/plugins/’);

By default (And you cant change this without a plugin) WordPress saves all your entered details in the database (except password, It only stores the server address, port number, and username), If you do this once, WordPress will remember them, You may then define your password in your wp-config.php file if you so wish. But be warned, In the case where your server is compromised, OR if other users on the system can read your files (As some badly setup shared hosts are), then you may be leaving yourself wide open to an attack if anyone ever feels like it.

Why doesnt WordPress support Method XYZ for acess?

Because no-one has contributed a patch to add support for it. In WordPress 2.5, SSH was not supported, this was only added in a later version (2.7) due to someone doing enough work on it for it to be included.

Why do you see yourself as the end-all for this?

Some people might be surprised to hear(And others will not be surprised) that i get this a lot, But i do. In short, I dont, Please, Go ahead and implement your ideas, I mearly voice my opinion, My opinion does not prevent something from being included, But due to my involvement, A yes from me, Might give a bit more weight to getting something in, The same goes for all the core developers, And all those of us (me included) who regularly contribute patches to WordPress.

WordPress is a comunity effort, Due to the size of the community though, There will always be some who are more well known than others, and thats unfortunate, But its impossible for core developers to keep track of everyone. Some people have said in the past that WordPress was being developed by a core few people who did what they wanted, and didnt take others thoughts into consideration. In some ways this is true, Matt for example, Has in the past said he didnt want something to happen, Due to the respect and the weight he carries, his voice might be the deciding vote, But plenty of times will there be things commited which Matt does not agree with. Many of the core developers develop code and features which users request, even if they have no need for it themselves, I would not be surprised if the top 20% of developers of WordPress only used 10% of its features. I personally for example, Have made very little use of the Plugin upgrader and installers.

So to wrap this item up, No. Just no. I will voice my opinion, However, Take it with a grain of salt, Just like everyone elses, If you want to rip every single contribution i’ve made to WordPress out, and replace it with something you see as better, DO IT, if its better, then great, WordPress will be better off. I will do the same thing, If i see someone submit a patch which i feel could be done in a better way, Then i’ll submit a alternate patch, Quite often it’ll be a “wow, i didnt realise it could be so simple” or a “that looks much cleaner!”, or even “Isnt that what my patch does????” Deal with it, And move on.

You forgot to answer something!

You know, I probably did, So if its before the end of March 09, Add a comment, and i’ll add it to the list. Why March? Quite simple really, After a month, i’m unlikely to care enough to update it. I might write an updated post in the future which corrects things here, or updates it for whatever is in wordpress at that point in time.

I hope this has been of some use to someone out there. If it has been, Then its served its purpose.

Week 12 Status update

Firstly i’d like to say, I’ve enjoyed Summer of Code, and thanks to the WP devs who chose to accept my proposal.
Secondly, i’d like to remind any testers that this plugin requires wordpress 2.3(currently in alpha).

The final Evaluation code is available in this branch: http://wordpress-soc-2007.googlecode.com/svn/trunk/dd32.crazyman/tags/gsoc-final-evaluation/

Whats happened since the mid week dash update?

Useful Things

This week the most useful resource i came accross was this blog entry: Timing is everything: scheduling in WordPress It outlines on how to use the WP-Cron functionality.

EDIT: Stupid Non-breaking spaces :(

Final Dash

So its the 17th today, GSoC pens down is officially 5am on tuesday(21st).

That means i’ve got this weekend basically to finish off everything. So what needs doing?

  1. Support for FTP other than via the inbuilt FTP Extension. This is semi done, Turns out the PemFTP library which i was relying on working doesnt seem to work at all for me (Strangely enough), but i’ve found another class PHP FTP which seems its going to work for me (Needs a lot of testing though).
  2. Upgrading of Plugins, this has been sitting there waiting to be done for awhile now, its one of the items i should’ve started with a lot earlier, but i kept putting it off to work on other items which it was to rely on.. Probably done 2/3 of that, just need to write the final copying code( It allready knows which files have changed, which have been deleted, whats new/etc, so it shouldnt be too hard)
  3. Bug Fixing.. Theres a lot more bugs surfacing, Mainly due to some functionality not being as streamlined as i wanted.
  4. Testing.. Theres not been too much testing in a variety of platforms, I was intending for this last part of the time to mainly be testing, but it doesnt look that way right now.
  5. Documentation, Luckely quite a lot of it is easy enough to follow in the files(IMHO), but i’m still going to need to write something down.
  6. Caching! Currently i make great use of the WordPress Obejct cache, however, As most people should know, Its not used for a LOT of wordpress installs. I Guess i need to throw some get_option/set_option in there for caching of the main output? (ie. the current status for the plugins(“Latest Installed” || “Update Available”) etc

What could’ve been done better.

This is something that most people will be writing after their evaluation is done, but hey, Its in the open anyway :)

  1. I came into GSoC not knowing much about the XMLRPC code which wordpress uses, In retrospect, i should’ve built the update server into that, and had it making requests via that for default, However thats not a great issue in my mind, Its only a few code changes in an extension file. At the same time i’d have liked to use RSS, but i didnt get time to locate a decent way to work with that(And representing 2d arrays in RSS to be parsed by a reader is impossible.)
  2. Time Management, I’ve never been good at time management, But i’m getting better..

Week 11 status report

Whats been acomplished this week.

  • Fixed Tag searching of WordPress.org
  • Made a lot of changes to the FTP-Ext class, Re-implemented a few functions which i wasnt happy about, added features to existing functions.
  • Made a move to $wp_update instead of $wpupdate to keep with WP coding
  • Made a move to a global $wp_filesystem class to make the coding simpler
  • Created a functionfolder_diff which relies on $wp_filesystem, used to create output such as this:
    dda-options.php (Changed)
    dda_portfolio.php (Changed)
    ddeviantart.php (Changed)
    inc (Changed)
    inc/dCustom.php (Changed)
    readme.txt (New)
    deleted_file (Deleted)
    same_file (unChanged)
  • Some more work on plugin upgrades
  • Many Many more small bugs which appeared during enhancements (And while testing on linux with FTP)

Plans for Next Week

  • Finally get a non-PHP-FTP-Extension FTP class implemented around the PemFTP classes.
  • More bugfixing (Just watch, more will come out of the woodwork)
  • More work on plugin upgrades, only another step or so left in that code i think.

The time left is Dwindling fast, incredibly fast actually, and uni workloads are increasing twice as fast.. I’m honestly supprised i got so much work done this week. (Yet so little with the time spent argueing with bugs)GSoC, WordPress, wp-update

Week 10 status report

Whats been acomplished this week.

  • More Filtering:
  • Few more external options
  • More Cleaning:
  • Fixed a few options
  • Droped the old theme installer and modified the plugin installation code to work with allmost anything
  • More Fiddling:
  • Started a Upgrade proceedure
  • Added Admin Notices and emailing of someone when the plugins need updating.

Plans for Next Week

  • More Cleaning
  • get a basic Upgrade working nicely
  • Work out wp-cron(For weekly checking of updates)
  • Might look into notifying that wordpress itself needs to be upgraded.

Week 9 status report

Whats been acomplished this week.

  • Tag Cloud for Plugin page
  • More filter work, which mainly consisted of moving all external access functionality into filters
  • Fixed up the options page for the Filesystem stuff
  • Initial planning of method to do upgrades (paper)
  • Better XML handling function for Wp-Update-Manager.

Plans for Next Week

  • Get back on the Install/Upgrade functions, At present theres a seperate function for plugins/themes, I’d like to roll this into a single function that can install anything.
  • Write an initial upgrade proceedure plugin, So far planning that it should show the files that have been modified/removed/untouched before allowing changes to me made.
  • Re-do any commenting on functionality thats changed (ie. search/upgrade functions)

Problems Encountered so far (And useful resources)

  • None of note.

Week 8 status report

Whats been acomplished this week.

  • Redesigning the Filesystem Options page.
  • Localised more strings
  • Improved Requirements Checking
  • Load jQuery via Add_action(This has made the plugin 2.3-only)
  • Prevented Prototype from loading on my pages(Fixes Incompatibilities when a plugin loads Prototype for the entire Admin console)
  • Started moving all external-access functions (Searching, Info, etc) into separate files and loading via filters/actions
  • [From trac patch] Hide edit column completely when plugin not editable
  • Re-designed some of the options page for filesystem.

Note: Not all of the above is checked into SVN yet, expect in the coming days.

Plans for Next Week

  • Finish more work on the options pages.

Problems Encountered so far (And useful resources)

  1. Visual jQuery

Uni semester is starting up again, Business units, PHP/Perl, Java, should be fun..