Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 1 of 2
PEAR Downloader and it’s share of faults……it’s time for some remedy or a change for good
 
SimpleHelixcom
Enthusiast
 
Avatar
Total Posts:  906
Joined:  2007-08-31
Huntsville, AL
 

I brought this up before and I’ll bring it up again.

I think we need to have a separate packages for servers that are either on mod_php and on fcgi/suExec (or replace PEAR completely and use some other options like FTP ) .

(Increasing amount of hosting companies are now on fcgi/suExec for security purposes, including all the leading major hosting companies currently at the moment)

The problem lies in the fact that mod_php and suExec treats permissions differently. If Magento did not have it’s PEAR downloader, we wouldn’t have any issues but the fact the PEAR downloader rewrites all permissions, makes this a very stringent problem.

We have to look at two facts:

1) mod_php will require files to have a chmod of 777 in order for PEAR downloader to work.
2) suExec doesn’t need any chmod settings, as it already runs process as the user. But since magento downloader chmods the files which are hardcoded in the script, it will break some hosting configuration and give out Internal 500 error.

Judging from this, i think it’d be wise if you guys either create a separate magento package or create a script that detects a mod_php or suExec environment so that for suExec it won’t touch the chmod settings at all.

If none of these options are going to be viable, then I guess we would need to switch to FTP to perform the downloads ala Drupal. This way, all files will be performed as the user and we would not be having this sort of a mess that we have now.

The fact that magento downloader out of the box chmods all files it touches to 777 is just plain bad security practice. And without SSH access, most users do not have the time to re-chmod the files back to their secured chmod values, especially those who use FTP to perform the CHMODs.

So I think during the Magento Installer, if it can give us an option to choose either mod_php or suexec, it would solve the problem for most of us but looking at the bigger picture, I’m starting to think PEAR is not the way, for mass public compatiblity issues.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Crucial
Enthusiast
 
Avatar
Total Posts:  770
Joined:  2007-11-07
Phoenix, AZ
 
SimpleHelix.com -

I think we need to have a separate packages for servers that are either on mod_php and on fcgi/suExec (or replace PEAR completely and use some other options like FTP ) .

I think it would be helpful to first understand the beauty of PEAR and the CLI so we can look at this from a programmer’s point of view, which will explain why it isn’t feasible for Magento to just get rid of PEAR for the sake of convenience.

Programmers love reusable code, it makes their life easier and ultimately easier for them to fix bugs and for others to develop new features. I guarantee you’re going to see PEAR being used more and more as PHP 5/6 and OOP practices become more mainstream.

Right now this isn’t the case. If you look at some of the major applications out there, like WordPress, Drupal, or vBulletin, they have outdated methods for adding plugins or extensions and upgrading software. They’ve all started to make the change towards a strictly PHP 5 level of programming, but because they’ve been around for a while, won’t be able to do this for at least a couple more years.

If you want to upgrade or install a plugin for one of these applications, you have to do it the way you mentioned, by manually downloading a compressed file, uploading it via FTP to your server, extracting the files, and then put everything in the correct place.

The result is something many people are familiar with if they’ve ever had to upgrade any of the aforementioned applications.

For example, try upgrading WordPress with 20 plugins and a couple themes installed. Not really what’d I’d call quick, easy, or trouble free. When you have to do this every other month when a new version is released, it’s very cumbersome and usually results in people not keeping their software updated. It’s even worse trying to keep track of the plugins you’ve installed and keeping those up to date too.

With no central repository or packaging structure, there will always be inconsistencies, bugs, or even some cases where an upgrade can render half the plugins you’ve installed useless.

PEAR, in terms of how it’s being used for Magento, is nothing but a distribution center for installing packages while also allowing for interpackage dependencies.

It takes away the hassle of upgrading an application or adding an extension. I’m looking forward to the day when I can upgrade WordPress with one command and not have to worry about manually replacing everything and trying to keep track of individual plugins.

Because Magento is strictly PHP 5 and relies on OOP and an MVC framework, it takes full advantage of PEAR. Those XML files you see all over the place are a prime example. Pick up any advanced PHP 5 or PHP 6 book and you’ll see a section talking about PEAR and the CLI.

SimpleHelix.com -

The problem lies in the fact that mod_php and suExec treats permissions differently. If Magento did not have it’s PEAR downloader, we wouldn’t have any issues but the fact the PEAR downloader rewrites all permissions, makes this a very stringent problem.

You can look at it another way: if Magento didn’t have a web-based interface for PEAR (Magento Connect), we’d eliminate even more problems. If everyone had SSH access they could use PEAR to upgrade Magento or install new extensions, and you would always be running as the correct user (instead of user nobody). By doing that, you eliminate the ownership and/or permissions issue. Now it doesn’t matter if you’re running suEXEC or not.

SimpleHelix.com -

mod_php will require files to have a chmod of 777 in order for PEAR downloader to work.

That’s only true if you’re using the web-based installer (Magento Connect). When PHP is setup in a non-suEXEC environment, all requests made from a browser will run as user nobody. So not only do you have permissions issues, but ownership issues. And even if the permissions are set, you still have to deal with the user nobody. This doesn’t apply to a suEXEC environment.

SimpleHelix.com -

The fact that magento downloader out of the box chmods all files it touches to 777 is just plain bad security practice. And without SSH access, most users do not have the time to re-chmod the files back to their secured chmod values, especially those who use FTP to perform the CHMODs.

You’ve mentioned this a couple times now, how Magento Connect changes all the permissions which can result in a 500 erorr for anyone running PHP as a CGI binary with suEXEC.

I don’t think this is the case, but just to be sure, I went ahead and installed Magento 1.1.2 with sample data on a server with PHP running as an Apache module (no suEXEC) and then on a server with PHP running as FastCGI (with suEXEC).

I did not set all of the files and directories to 777 for the Magento installation that was running in the non-suEXEC environment. I followed the instructions listed on this site. So both installations were exactly the same, except the non-suEXEC one did not need the php.ini file.

I made sure everything worked, and then upgraded the Magento installation on the non-suEXEC server via SSH, and then upgraded Magento on the FastCGI server via Magento Connect. I repeated the steps for the FastCGI install again, but used Magento Connect this time.

Note: You have to use SSH to install on a non-suEXEC environment because the upgrade process needs to be handled by the actual user who owns the files. If you do it via Magento Connect, it runs as user nobody, and that’s where problems start to happen. I believe you can actually do one upgrade from Magento Connect, but once you’ve done an upgrade or extension install, you won’t be able to properly use Magento Connect again to perform upgrades.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Crucial
Enthusiast
 
Avatar
Total Posts:  770
Joined:  2007-11-07
Phoenix, AZ
 

(continued...)

Each installation and upgrade worked as planned. There were no errors during or after the upgrade, and none of the file permissions were changed. Files were left as 644, directories as 755. I even compared them before and after just to be sure.

I’m unable to replicate the issue you’re having though, so if you could go in to detail on how I could do this that would be great. I also don’t believe Magento changes all of the files and directories to 777. I’ve performed countless upgrades on suEXEC and non-suEXEC environments and haven’t come across this issue yet.

It’s up to the programmers to make sure the application works in various environments, but it’s ultimately up to the person installing or upgrading the application to know the correct way to do this.

I don’t think getting rid of PEAR or making multiple packages is a good idea. Magento would be going backwards as an application. Not to mention it would most likely require changes on a core level.

And no matter what application we’re talking about, there will always be issues because of the number of ways you can setup a server. Different operating systems, different PHP and MySQL versions, missing extensions, different handlers for PHP (module, CGI, FastCGI, suPHP), suEXEC or non-suEXEC, etc.

If you go to a community forum for any of the major applications out there, the most active threads will always be for installing and upgrading said application and dealing with operational errors. Tthat’s just the nature of software development.

 
Magento Community Magento Community
Magento Community
Magento Community
 
SimpleHelixcom
Enthusiast
 
Avatar
Total Posts:  906
Joined:  2007-08-31
Huntsville, AL
 

@crucial,

Although I do agree with your point of view into the OOP patterns of php, unfortunately I don’t think PEAR is any help in to the trend to php 5 OOP.
PEAR was available since the back in PHP 4 days, it’s really got nothing to do with what I was trying to mention.

And Magento itself really is just using PEAR as a file distribution center like you said, so there really isn’t any reason to be using PEAR when there are other better alternatives such as FTP or even sFTP. It’d work exactly the same as PEAR, except that the files will be strictly owned as “user” rather than the nobody/apache, removing any need for chmod 777 settings anywhere.

As for your installation, I wonder if you used Magento Downloader to do the install?
I mainly use Magento Downloader as that gets all the software up-to-date and ready so I’m not sure if this would be the case .....

But aside from all that, look at the forum and view all these people needing help on how to either upgrade nor how to even install. These should be relatively straight-forward simple tasks. But because of the issues we have over the file permission/ownerships, they now have a problem on their hands that they probably do not have any idea about,..... if their platform is suExec or non-suExec. These guys also want to do it themselves, as they are developers, and a lot of them do not have the knowledge of SSH to perform any of these tasks adequately.

So I figure that we can probably avoid all these issues if we just take care of the transport method so all the files will always be owned as the user.

IF CLI is your thing, Magento also has options for SVN, which should be enough to perform all types of upgrades, etc....which would be another reason why PEAR won’t be needed. And as developers, I’m pretty sure we will prefer SVN/CVS so we can do things like diff and compare, as well as keep a log of all the things that happened in-between each updates, and importantly, the ability to roll-back whenever there is a problem. But just letting PEAR do it’s thing is a step downgrade towards programming, which becomes kind of like jumping off a cliff with your eyes blind-folded.

Upgrades also are dangerous because it affects the database layer as well, a half-finished upgrade will likely give you a broke database in which you have to fix manually script by script.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Crucial
Enthusiast
 
Avatar
Total Posts:  770
Joined:  2007-11-07
Phoenix, AZ
 
SimpleHelix.com -

As for your installation, I wonder if you used Magento Downloader to do the install?
I mainly use Magento Downloader as that gets all the software up-to-date and ready so I’m not sure if this would be the case .....

I’ve never used the downloader version actually, but if that’s what you are using, I’ll run some tests with that method. I prefer the full version because I also initialize the PEAR registry, which upgrades the installation to the latest version. When that’s finished, then I can proceed with the web-based installer. Well, correction. I used to use the web-based installer. Instead, I do it through SSH.

SimpleHelix.com -

But aside from all that, look at the forum and view all these people needing help on how to either upgrade nor how to even install. These should be relatively straight-forward simple tasks.

I agree. I guess the question should be raised, is it realistic to expect the average person to perform an upgrade? Or is this a case where you really need a third-party to do this?

I would say it’s not unrealistic. The level of skill needed to download, upload, and extract files with FTP software is pretty similar in terms of skill level, but I think peole are being introduced to concepts that they’ve never had to deal with (like upgrading from your web browser or connecting to SSH).

And actually, I think if you could give someone an SSH “crash course,” they’d find it much easier to do things. For example, I bet most people install Magento like this:

1. Download the ZIP file and the 50MB+ sample data file.
2. Save it locally to your computer.
3. Open the ZIP file up and extract the files.
4. Connect to your site via FTP and upload the extracted files.
5. Set the file permissions using your FTP client.
6. Set up a blank database and assign a user to it using your web control panel.
7. Proceed with the web-based installer.

And if that’s how most people are doing installs, you also have to think about how many aren’t even using FTP, but are instead using the File Manager application that most web control panels have.

Most people “get” the traditional way that you mentioned, probably because it’s how things have been done for so long. So I can understand that change is hard and concepts like using SSH are completely out of the question.

Ultimately, I just think there needs to be more clear documentation on this. That goes for installing Magento and upgrading it. There should be two separate set of instructions depending on your PHP environment.

SimpleHelix.com -

So I figure that we can probably avoid all these issues if we just take care of the transport method so all the files will always be owned as the user.

But from what I’m seeing, this really only affects non-suEXEC users, which you said is not how most hosts run. In a suEXEC environment, we don’t get these issues (with the exception of the one you’re talking about which I will have to test out later). But putting your issue aside for a moment, suEXEC does eliminate these permissions issue, correct?

SimpleHelix.com -

And as developers, I’m pretty sure we will prefer SVN/CVS so we can do things like diff and compare, as well as keep a log of all the things that happened in-between each updates. But just letting PEAR do it’s thing is a step downgrade towards programming, which becomes kind of like a blind leading a blind in most cases, as there are times when your magento will break after upgrades.

I suppose it comes down to what you’re comfortable with. I personally don’t use SVN, and find the PEAR method much easier.

But I think you’re looking at this from the wrong perspective. What Magento Connect aims to do is replace the traditional method of upgrading software. Instead, it’s letting users upgrade from their browser.

If SSH is a foreign concept to some people, then it also stands to reason that FTP is also foreign for a lot of people. So even if Magento Connect didn’t exist, and all we had was SVN (for those that knew SSH) and the traditional method, we’d still have people who had trouble upgrading Magento.

SimpleHelix.com -

Upgrades also are dangerous because it affects the database layer as well, a half-finished upgrade will likely give you a broke database in which you have to fix manually script by script.

Yeah, upgrades are always risky. I’m sure you tell your clients the same thing we do though, to always make a backup before making any changes. There’s certainly things we can do to avoid problems if they do come up.

Anyways, I understand there’s frustration from both ends, but I really think some clear documentation would alleviate a lot of problems. You and I both know the documentation for Magento isn’t the best, and certainly not setup like other online documentations are. New members to the community have a hard time even finding it, and when they do, might get conflicting or incomplete information, or the instructions are correct for “X” environment, but not “Y.”

 
Magento Community Magento Community
Magento Community
Magento Community
 
misteroriginal
Member
 
Total Posts:  48
Joined:  2008-03-26
 

Holy cow! You guys talk a lot. Too much to comment on, so let me pick one little bitty thing.

The fact that magento downloader out of the box chmods all files it touches to 777 is just plain bad security practice.

Amen!

But I wonder about the whole architecture. Seems like some of the file permission issues could be idiot-proofed if the lib and app folders were above public_html.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Crucial
Enthusiast
 
Avatar
Total Posts:  770
Joined:  2007-11-07
Phoenix, AZ
 
misteroriginal - 18 September 2008 10:18 PM

But I wonder about the whole architecture. Seems like some of the file permission issues could be idiot-proofed if the lib and app folders were above public_html.

I’m actually wondering why they didn’t do an approach like you’re describing. Most MVC frameworks I’ve worked with let you put all of the files above the public_html directory, leaving only the index.php “routing” file in the public_html directory, along with anything else that needs to be public (like images).

I’m not sure if that would solve permissions issues though, regardless of where the directories are, it would still have to write to them.

Also, your comment about the permissions appears to only apply to suPHP, because I tested and confirmed that this wasn’t happening when PHP was a DSO or running through FastCGI, and I’ve been doing upgrades on both types of PHP environments for a while now and haven’t encountered this problem.

 
Magento Community Magento Community
Magento Community
Magento Community
 
misteroriginal
Member
 
Total Posts:  48
Joined:  2008-03-26
 
Crucial - 19 September 2008 07:38 PM

I’m not sure if that would solve permissions issues though, regardless of where the directories are, it would still have to write to them.

I’m just thinking if the permissions on the files and directories are set to 777, it’s better to NOT have them publicly accessible. It doesn’t solve the problem, but I thought it might be a little more secure regardless the mod_php / suExec issue.

Generally speaking ...
Software products are ‘sold’ based on feature lists and ease of use. Security is always passed off as the ultimate responsibility of the end-user. For hosts of web apps, I can imagine some frustration, but it seems to me that it would be easier for the host to write detailed instructions on how to make sure a given app is securely installed than for the developer to try to detect every possible server configuration or misconfiguration. It boils down to responsibility, or even liability.

The developer is responsible for making the application fundamentally secure. The host is responsible for making the hosting environment fundamentally secure. The end-user is ultimately responsible for the application. The host can help the end-user understand the specific needs of the environment better than the developer can write an auto-detect script or delete a feature that helps ‘sell’ the app.

I can’t imagine hosting Magento through a provider who does not provide shell access.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Crucial
Enthusiast
 
Avatar
Total Posts:  770
Joined:  2007-11-07
Phoenix, AZ
 
misteroriginal - 20 September 2008 04:39 AM

I’m just thinking if the permissions on the files and directories are set to 777, it’s better to NOT have them publicly accessible. It doesn’t solve the problem, but I thought it might be a little more secure regardless the mod_php / suExec issue.

Ah, you’re right. I wasn’t thinking straight, been a hectic week wink

misteroriginal - 20 September 2008 04:39 AM

but it seems to me that it would be easier for the host to write detailed instructions on how to make sure a given app is securely installed than for the developer to try to detect every possible server configuration or misconfiguration. It boils down to responsibility, or even liability.

Great point actually regarding the hosts having instructions. Noted.

misteroriginal - 20 September 2008 04:39 AM

I can’t imagine hosting Magento through a provider who does not provide shell access.

I personally can’t imagine signing up for hosting that didn’t have SSH either, regardless of if it was just for Magento or not. As a designer/developer myself, it just makes things so much easier.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Unirgy
Guru
 
Avatar
Total Posts:  478
Joined:  2007-09-07
 
Crucial - 19 September 2008 07:38 PM

misteroriginal - 18 September 2008 10:18 PM
But I wonder about the whole architecture. Seems like some of the file permission issues could be idiot-proofed if the lib and app folders were above public_html.

I’m actually wondering why they didn’t do an approach like you’re describing. Most MVC frameworks I’ve worked with let you put all of the files above the public_html directory, leaving only the index.php “routing” file in the public_html directory, along with anything else that needs to be public (like images).

That should be feasible, see Mage::run declaration:

public static function run($code ''$type 'store'$options=array())
There’s this mysterious $options argument which after some research in Mage_Core_Model_Config_Options reveals interesting things, like:
Mage::run('default''store', array('app_dir'=>'/home/user/magento/app''lib_dir'=>'/home/user/magento/lib'));
I wonder if that would work smile
 
Magento Community Magento Community
Magento Community
Magento Community
 
Unirgy
Guru
 
Avatar
Total Posts:  478
Joined:  2007-09-07
 

Now, regarding the mode 0777 issue, it affects only PHP files that are being spawned from Apache, namely:

./index.php
./downloader/index.php
./js/index.php
./report/index.php

It would be fairly easy to make a script that would restore 0755 for these files on suPHP hosts.

 
Magento Community Magento Community
Magento Community
Magento Community
 
SimpleHelixcom
Enthusiast
 
Avatar
Total Posts:  906
Joined:  2007-08-31
Huntsville, AL
 

Yeah I said that all along about why Magento’s core files aren’t outside the public root folder.

There is just no need for that, unless Varien felt it was complicating things for the mass users out there, which I’d understand because it’d be pretty hard to explain all the x’s and o’s just to get the magento up and running for the new users.

But this is something that should ideally be practiced in a production environment and I believe is the way the future holds for PHP.

 
Magento Community Magento Community
Magento Community
Magento Community
 
SimpleHelixcom
Enthusiast
 
Avatar
Total Posts:  906
Joined:  2007-08-31
Huntsville, AL
 

First of all, it’s good to read all the feedbacks. From looking at it all, I just sum up it as that, there are fixes and practices that need to be placed and possibly warned on many types of hosting platforms. A documentation on this would be good but then again, it’d be fairly confusing for the end-user at first glance.

Because of the chmod issues, well this problem is there if you use “Magento Downloader to install your Magento”, regardless of whether you are on mod_php/suExec/suPHP, as all the files will be reset to the improper chmod values on the initial run.

This is why we have this conflicts on proper way to install Magento, as ideally, all the methods should really give you the same results.

You have to expect and understand that most of users will probably install Magento through Magento Downloader, mainly to save bandwidth and save time.

But because this is not fully compatible with all types of hosting environment, I believe we have a bit of a problem as this raises concerns on security more than anything with some other hosting companies that is not properly configured for this.

You can expect everything to run well if you install Magento and upgrade Magento using SSH but I think users who use SSH in general is a very small number and most will want to do it all from the Admin ... to perform things like upgrades, etc....

So although using SSH to install Magento is probably the most preferred option (at this point), expecting users to learn and use SSH is not really all that realistic.

Once users gets hooked to SSH, they would be performing all types of stuff, delete/create/edit files and folders, and as they are still learning, they’d undoubtedly get into a mistake that they probably can’t go back such as deleting an entire folder tree. (believe me, we’ve had many types of these situations from our clients)

Anyhow, i just wanted to know what Varien thinks about moving file transfer method from PEAR to FTP. This has been tested to work fine on Drupal and I think it’s a good method for these type of purposes. Switching to FTP, this would require user to enter in their ftp login information on the installation phase, which I don’t think it’s all that bad an idea. I think Varien can also setup their own proper repository and code so the proper files are distributed across. And doing this, would solve all the issues we discussed so far.

So I guess we come full circle now, and back to my initial question, what do you guys think of this switch to ftp, or possibly adding this alongside as another alternative method to PEAR?

 
Magento Community Magento Community
Magento Community
Magento Community
 
Fontis
Sr. Member
 
Avatar
Total Posts:  99
Joined:  2007-08-31
Melbourne, Australia
 

I think the FTP method is feasible - Wordpress 2.6 uses FTP to allow automated upgrade of plugins from the admin panel. You basically just need to provide your FTP username and password and the latest version of the plugin is downloaded and installed using the correct user credentials from the central Wordpress plugin repository. No (direct) messing about with either SSH or an FTP client.

I don’t see why a similar system wouldn’t work for Magento as well. In fact, I see no reason why they couldn’t take it further and allow you to browse and install extensions directly from your admin panel. This would be more convenient that browsing on Connect, getting the install key then entering it into either your Connect Manager interface or into PEAR using SSH.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Unirgy
Guru
 
Avatar
Total Posts:  478
Joined:  2007-09-07
 

@Chris, just a sec, i’m not sure i understood you correctly, do you mean you enter your personal server’s FTP credentials into WordPress central database?

 
Magento Community Magento Community
Magento Community
Magento Community
 
Unirgy
Guru
 
Avatar
Total Posts:  478
Joined:  2007-09-07
 

No, you’ve probably meant you provide FTP user/pass in your WordPress installation, so it will upload locally using FTP.

Sorry, just understood what you wrote incorrectly.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 1 of 2