Compare commits

...
5 Commits
27 changed files with 394 additions and 117 deletions
+37
View File
@@ -4,6 +4,43 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
## [0.20.1] - 2026-09-24
### Fixed
- `Modules\Core\Catalog\Support\ProductDocumentLocalizer::withLocalizedFields()` — a translated
attribute (name, description, ...) saved blank for the current locale kept the empty string
instead of falling back to the store's default language, since `??` only falls back on a
missing/null key, not an empty one. A product with no English copy yet showed a blank
title/description on `/en/` instead of its Greek content.
- `Modules\Core\Command\WipeCatalogCommand` — now also deletes every `CartLine` referencing a
`product_variant` purchasable as part of the wipe (line items only, `Cart` records themselves
are left alone). Previously, any cart still holding a line for a wiped variant crashed the
entire storefront on every page load (`PricingManager::for()` throws when the variant a line
points at no longer exists) until those dangling lines were removed by hand.
- `Modules\Core\Command\MigrateImportCommand` now asks which language a Shopify export file's
own text is written in before importing, instead of silently assuming it matches the store's
default language — the two are independent facts, and a mismatch used to save every imported
product's name/description under the wrong language. Backing class renamed `DefaultLocale` →
`Modules\Core\MigrateImport\Services\ImportLocale` to stop implying that assumption.
### Changed
- `Modules\Core\MigrateImport` reorganized to match every other module's layout
(`Contracts/`, `DTOs/`, `Jobs/`, `Models/`, `Services/`) instead of loose files at each
namespace root — no behavior change, but every `use` of `Importer`, `ImportSpec`,
`ImporterFactory`, `ImportLocale`, `RunMigrateImportJob`, `ShopifyExportImporter`,
`ShopifyCsvReader`, `ProductGroup`, `JudgeMeExportImporter`, and `JudgeMeCsvReader` moved to
its new namespace.
- `Modules\Core\MigrateImport\Shopify\Services\ShopifyExportImporter::import()` now dispatches
one `Jobs\ImportShopifyProductJob` per product (via `Bus::batch()`) instead of importing every
product inline in a single queued job. A large export's variants, resolvers, and media
downloads accumulating in one long-lived process routinely exceeded `queue:work`'s
`--memory` limit; the worker died mid-run, the container restarted, and the entire import
started over from the first row every time, never actually finishing. Splitting into one job
per product resets memory between products, and a restart now only repeats whichever single
product was in flight. `Modules\Core\Catalog\Services\SkuBackfillService::backfill()` moved
from running right after the import loop to the batch's `then()` callback, since it must wait
for every product job to finish rather than firing the moment jobs are merely queued.
## [0.20.0] - 2026-09-23 ## [0.20.0] - 2026-09-23
### Added ### Added
+1 -1
View File
@@ -2,7 +2,7 @@
"name": "boboko/core", "name": "boboko/core",
"description": "Core module — authentication and shared panel behaviour", "description": "Core module — authentication and shared panel behaviour",
"type": "library", "type": "library",
"version": "0.20.0", "version": "0.20.1",
"autoload": { "autoload": {
"psr-4": { "psr-4": {
"Modules\\Core\\": "src/" "Modules\\Core\\": "src/"
+4 -3
View File
@@ -8,15 +8,16 @@ use Lunar\Models\ProductVariant;
* Generates a SKU for every ProductVariant missing one — extracted out of * Generates a SKU for every ProductVariant missing one — extracted out of
* Command\BackfillMissingSkusCommand (which becomes a thin CLI wrapper * Command\BackfillMissingSkusCommand (which becomes a thin CLI wrapper
* around this, keeping --dry-run/progress-bar concerns out of the * around this, keeping --dry-run/progress-bar concerns out of the
* reusable logic) so MigrateImport\RunMigrateImportJob can also call it * reusable logic) so MigrateImport\Shopify\Services\ShopifyExportImporter can
* directly, right after a Shopify import, with no CLI concerns at all. * also call it directly, once every product job in its import batch has
* finished (see that class's own import()), with no CLI concerns at all.
* *
* Format is "SKU-P{product_id}-V{variant_id}": deterministic and * Format is "SKU-P{product_id}-V{variant_id}": deterministic and
* guaranteed unique without a uniqueness check, since product_id/ * guaranteed unique without a uniqueness check, since product_id/
* variant_id already are. Only variants with a null `sku` are touched — * variant_id already are. Only variants with a null `sku` are touched —
* not an importer bug when one shows up after a Shopify import, the * not an importer bug when one shows up after a Shopify import, the
* source CSV rows genuinely had no `Variant SKU` value (see * source CSV rows genuinely had no `Variant SKU` value (see
* MigrateImport\Shopify\ShopifyExportImporter). * MigrateImport\Shopify\Services\ShopifyExportImporter).
*/ */
class SkuBackfillService class SkuBackfillService
{ {
@@ -51,7 +51,15 @@ class ProductDocumentLocalizer
$availableLocales = $this->languages->availableLocales(); $availableLocales = $this->languages->availableLocales();
foreach ($this->translatedAttributeHandles() as $handle) { foreach ($this->translatedAttributeHandles() as $handle) {
$product[$handle] = $product[$handle.'_'.$locale] ?? $product[$handle.'_'.$fallbackLocale] ?? null; // filled(), not ?? - a translated attribute saved blank for
// the current locale still has that {handle}_{locale} key in
// the document, just set to '' rather than absent. ?? only
// falls back on a missing/null key, so it kept the empty
// string instead of falling through to a locale that actually
// has content.
$product[$handle] = filled($product[$handle.'_'.$locale] ?? null)
? $product[$handle.'_'.$locale]
: ($product[$handle.'_'.$fallbackLocale] ?? null);
foreach ($availableLocales as $availableLocale) { foreach ($availableLocales as $availableLocale) {
unset($product[$handle.'_'.$availableLocale]); unset($product[$handle.'_'.$availableLocale]);
+1 -1
View File
@@ -10,7 +10,7 @@ use Modules\Core\Catalog\Services\SkuBackfillService;
* CLI wrapper (--dry-run, a progress bar) around Catalog\Services\ * CLI wrapper (--dry-run, a progress bar) around Catalog\Services\
* SkuBackfillService — see that class's own docblock for the actual * SkuBackfillService — see that class's own docblock for the actual
* backfill logic, also called automatically after a Shopify import (see * backfill logic, also called automatically after a Shopify import (see
* MigrateImport\RunMigrateImportJob). * MigrateImport\Jobs\RunMigrateImportJob).
*/ */
class BackfillMissingSkusCommand extends Command class BackfillMissingSkusCommand extends Command
{ {
+41 -2
View File
@@ -3,8 +3,9 @@
namespace Modules\Core\Command; namespace Modules\Core\Command;
use Illuminate\Console\Command; use Illuminate\Console\Command;
use Modules\Core\MigrateImport\ImportSpec; use Lunar\Models\Language;
use Modules\Core\MigrateImport\RunMigrateImportJob; use Modules\Core\MigrateImport\DTOs\ImportSpec;
use Modules\Core\MigrateImport\Jobs\RunMigrateImportJob;
class MigrateImportCommand extends Command class MigrateImportCommand extends Command
{ {
@@ -62,11 +63,29 @@ class MigrateImportCommand extends Command
$credentials = null; $credentials = null;
} }
// Shopify's own product export is a flat CSV — one Title/Body
// (HTML)/etc. column per row, no per-locale columns at all — so
// its text is necessarily written in exactly one language, and
// there is no reliable way to detect which one from the file
// itself. Modules\Core\MigrateImport\Services\ImportLocale::code() used to
// (as its former name, DefaultLocale, admits) assume it always
// matched this store's own Lunar\Models\
// Language::getDefault(), which is often wrong (a store's default
// admin/storefront language and the language a given export
// happens to be written in are two independent facts) — every
// imported product's name/description then saved silently under
// the wrong language, invisible unless that language happened to
// also be selected when viewing/editing the product afterward.
$locale = $source === 'shopify' && $type === 'export'
? $this->askImportLocale()
: null;
$spec = new ImportSpec( $spec = new ImportSpec(
source: $source, source: $source,
type: $type, type: $type,
filePath: $filePath, filePath: $filePath,
credentials: $credentials, credentials: $credentials,
locale: $locale,
); );
RunMigrateImportJob::dispatch($spec); RunMigrateImportJob::dispatch($spec);
@@ -74,6 +93,26 @@ class MigrateImportCommand extends Command
$this->info('Import queued.'); $this->info('Import queued.');
} }
/**
* Choices come from Language::all() — the same list an admin manages
* from the Filament panel (Settings > Languages) — not a hardcoded
* set, so a language this store doesn't have yet simply isn't
* offered here; the hint below says where to add it instead of this
* command silently accepting an arbitrary code Lunar has no row for.
*/
private function askImportLocale(): string
{
$languages = Language::orderBy('default', 'desc')->get(['code', 'name']);
return $this->choice(
"Which language is the export file's own text (product titles, descriptions, etc.) written in?\n".
' (Not necessarily this store\'s default language — the two are independent. '.
"If the language you need isn't listed, add it first from the admin panel under Languages.)",
$languages->mapWithKeys(fn (Language $language) => [$language->code => "{$language->name} ({$language->code})"])->all(),
$languages->first()?->code,
);
}
// Answers are relative to storage/app/private/imports (e.g. "shopify" or // Answers are relative to storage/app/private/imports (e.g. "shopify" or
// "shopify/products_export.csv"); absolute paths are used as-is. A // "shopify/products_export.csv"); absolute paths are used as-is. A
// directory answer picks the first CSV file found inside it. // directory answer picks the first CSV file found inside it.
+18 -4
View File
@@ -3,6 +3,7 @@
namespace Modules\Core\Command; namespace Modules\Core\Command;
use Illuminate\Console\Command; use Illuminate\Console\Command;
use Lunar\Models\CartLine;
use Lunar\Models\Product; use Lunar\Models\Product;
use Modules\Core\Auth\Models\Staff; use Modules\Core\Auth\Models\Staff;
use Modules\Core\Auth\Services\OtpService; use Modules\Core\Auth\Services\OtpService;
@@ -15,9 +16,10 @@ use function Laravel\Prompts\text;
* Irreversibly deletes every Product and everything that only exists * Irreversibly deletes every Product and everything that only exists
* because of a product — variants, variant prices, product-option value * because of a product — variants, variant prices, product-option value
* assignments, product images/media, product associations, the * assignments, product images/media, product associations, the
* ImportMapping rows tying them back to an external source, and the * ImportMapping rows tying them back to an external source, product-
* Meilisearch product index. Deliberately does NOT touch catalog * variant CartLine rows (line items only — Cart records themselves are
* STRUCTURE other products could still reference: ProductOption/ * left alone), and the Meilisearch product index. Deliberately does NOT
* touch catalog STRUCTURE other products could still reference: ProductOption/
* ProductOptionValue definitions ("Size", "Color" as reusable option * ProductOptionValue definitions ("Size", "Color" as reusable option
* types), Brands, Collections, Tags, Customer Groups — none of those are * types), Brands, Collections, Tags, Customer Groups — none of those are
* products, they're config a merchant would otherwise have to rebuild * products, they're config a merchant would otherwise have to rebuild
@@ -140,7 +142,7 @@ class WipeCatalogCommand extends Command
* for the skipped products) whose 'image' ImportMapping rows then * for the skipped products) whose 'image' ImportMapping rows then
* caused a LATER Shopify re-import to silently reuse those now- * caused a LATER Shopify re-import to silently reuse those now-
* orphaned Media objects instead of importing fresh ones — see * orphaned Media objects instead of importing fresh ones — see
* MigrateImport\Shopify\ShopifyExportImporter::resolveOrImportImage()'s * MigrateImport\Shopify\Services\ShopifyExportImporter::resolveOrImportImage()'s
* own docblock for that half of the same incident. Always re-querying * own docblock for that half of the same incident. Always re-querying
* the first N remaining rows (never advancing an id cursor) guarantees * the first N remaining rows (never advancing an id cursor) guarantees
* every product is actually visited exactly once, however many are * every product is actually visited exactly once, however many are
@@ -150,6 +152,18 @@ class WipeCatalogCommand extends Command
{ {
ImportMapping::whereIn('source_type', ['product', 'variant', 'image'])->delete(); ImportMapping::whereIn('source_type', ['product', 'variant', 'image'])->delete();
// Only the line items — not the parent Cart rows. This command is
// meant for early-stage/setup use where no real customer carts
// matter yet, but a customer's Cart record also anchors their
// session/coupon/address state; deleting it outright is more than
// "the catalog is gone" calls for. Leaving every variant a cart
// line could reference about to be force-deleted below would
// otherwise reproduce the exact storefront crash this step exists
// to prevent: CartLine::purchasable() resolves to null,
// PricingManager::for() throws a TypeError on every page load that
// renders the cart drawer.
CartLine::where('purchasable_type', 'product_variant')->delete();
while (true) { while (true) {
// withTrashed(): Product/ProductVariant both use SoftDeletes // withTrashed(): Product/ProductVariant both use SoftDeletes
// — a plain query would stop seeing a product the moment // — a plain query would stop seeing a product the moment
+10
View File
@@ -0,0 +1,10 @@
<?php
namespace Modules\Core\MigrateImport\Contracts;
use Modules\Core\MigrateImport\DTOs\ImportSpec;
interface Importer
{
public function import(ImportSpec $spec): void;
}
+25
View File
@@ -0,0 +1,25 @@
<?php
namespace Modules\Core\MigrateImport\DTOs;
class ImportSpec
{
/**
* @param ?string $locale The language the export file's own text
* (product titles, descriptions, option names, ...) is actually
* written in — asked of the operator at import time (see
* Command\MigrateImportCommand), since an export has no reliable
* way to declare its own language and it does not necessarily match
* this store's Lunar\Models\Language::getDefault(). Null for a
* source/type this doesn't apply to (e.g. an API-based import with
* no free-text file to attribute a single language to).
*/
public function __construct(
public readonly string $source,
public readonly string $type,
public readonly ?string $filePath = null,
public readonly ?array $credentials = null,
public readonly ?string $locale = null,
) {
}
}
-13
View File
@@ -1,13 +0,0 @@
<?php
namespace Modules\Core\MigrateImport;
use Lunar\Models\Language;
class DefaultLocale
{
public static function code(): string
{
return Language::getDefault()->code;
}
}
-14
View File
@@ -1,14 +0,0 @@
<?php
namespace Modules\Core\MigrateImport;
class ImportSpec
{
public function __construct(
public readonly string $source,
public readonly string $type,
public readonly ?string $filePath = null,
public readonly ?array $credentials = null,
) {
}
}
-8
View File
@@ -1,8 +0,0 @@
<?php
namespace Modules\Core\MigrateImport;
interface Importer
{
public function import(ImportSpec $spec): void;
}
@@ -0,0 +1,45 @@
<?php
namespace Modules\Core\MigrateImport\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;
use Modules\Core\MigrateImport\DTOs\ImportSpec;
use Modules\Core\MigrateImport\Services\ImporterFactory;
class RunMigrateImportJob implements ShouldQueue
{
use Dispatchable;
use InteractsWithQueue;
use Queueable;
use SerializesModels;
public function __construct(
public readonly ImportSpec $spec,
) {
}
/**
* Just hands off to the right Importer and returns — for Shopify,
* that importer dispatches a job batch instead of importing inline
* (see Shopify\ShopifyExportImporter::import()) and this job's own
* work is done the moment that batch is queued, well before the
* batch's jobs actually run. The SKU backfill that used to happen
* right here, after import() returned, now happens in that batch's
* own then() callback instead — running it here would fire before a
* single product had actually been imported.
*/
public function handle(): void
{
Log::info('Import started', ['source' => $this->spec->source, 'type' => $this->spec->type, 'file' => $this->spec->filePath]);
$importer = ImporterFactory::make($this->spec);
$importer->import($this->spec);
Log::info('Import job complete', ['source' => $this->spec->source]);
}
}
@@ -3,7 +3,6 @@
namespace Modules\Core\MigrateImport\JudgeMe\Resolvers; namespace Modules\Core\MigrateImport\JudgeMe\Resolvers;
use Lunar\Models\Product; use Lunar\Models\Product;
use Lunar\Models\Url;
class ProductResolver class ProductResolver
{ {
@@ -1,6 +1,6 @@
<?php <?php
namespace Modules\Core\MigrateImport\JudgeMe; namespace Modules\Core\MigrateImport\JudgeMe\Services;
use RuntimeException; use RuntimeException;
@@ -1,12 +1,12 @@
<?php <?php
namespace Modules\Core\MigrateImport\JudgeMe; namespace Modules\Core\MigrateImport\JudgeMe\Services;
use Throwable; use Throwable;
use Illuminate\Support\Carbon; use Illuminate\Support\Carbon;
use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log;
use Modules\Core\MigrateImport\Importer; use Modules\Core\MigrateImport\Contracts\Importer;
use Modules\Core\MigrateImport\ImportSpec; use Modules\Core\MigrateImport\DTOs\ImportSpec;
use Modules\Core\MigrateImport\JudgeMe\Resolvers\ProductResolver; use Modules\Core\MigrateImport\JudgeMe\Resolvers\ProductResolver;
use Modules\Core\MigrateImport\Models\ImportMapping; use Modules\Core\MigrateImport\Models\ImportMapping;
use Modules\Core\Review\Models\ProductReview; use Modules\Core\Review\Models\ProductReview;
@@ -22,11 +22,24 @@ class JudgeMeExportImporter implements Importer
public function import(ImportSpec $spec): void public function import(ImportSpec $spec): void
{ {
foreach ($this->csvReader->read($spec->filePath) as $row) { $rows = $this->csvReader->read($spec->filePath);
$total = count($rows);
$imported = 0;
Log::info('JudgeMe import: starting', ['total' => $total]);
foreach ($rows as $row) {
$this->importReview($row); $this->importReview($row);
$imported++;
if ($imported % 100 === 0) {
Log::info('JudgeMe import: progress', ['imported' => $imported, 'total' => $total]);
} }
} }
Log::info('JudgeMe import: finished', ['imported' => $imported, 'total' => $total]);
}
private function importReview(array $row): void private function importReview(array $row): void
{ {
$handle = trim((string) ($row['product_handle'] ?? '')); $handle = trim((string) ($row['product_handle'] ?? ''));
-39
View File
@@ -1,39 +0,0 @@
<?php
namespace Modules\Core\MigrateImport;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Modules\Core\Catalog\Services\SkuBackfillService;
class RunMigrateImportJob implements ShouldQueue
{
use Dispatchable;
use InteractsWithQueue;
use Queueable;
use SerializesModels;
public function __construct(
public readonly ImportSpec $spec,
) {
}
public function handle(SkuBackfillService $skuBackfill): void
{
$importer = ImporterFactory::make($this->spec);
$importer->import($this->spec);
// Only Shopify's importer creates ProductVariant rows at all (see
// Shopify\ShopifyExportImporter) — JudgeMe never touches products,
// so running this for that source would just be a guaranteed
// no-op query every time. Source CSV rows genuinely can have no
// `Variant SKU` value; see SkuBackfillService's own docblock for
// why that's synthesized rather than treated as an importer bug.
if ($this->spec->source === 'shopify') {
$skuBackfill->backfill();
}
}
}
@@ -0,0 +1,52 @@
<?php
namespace Modules\Core\MigrateImport\Services;
use Lunar\Models\Language;
/**
* The language every resolver in an import run writes product-facing text
* under (name, description, option names, ...) — NOT necessarily this
* store's own Lunar\Models\Language::getDefault(). An export file has no
* reliable way to declare its own language, and a store's default admin/
* storefront language is an independent fact from whatever language a
* given export happens to be written in — conflating the two (this
* class's own former name, DefaultLocale, said as much) used to silently
* save every imported product's text under the wrong language, invisible
* unless that language was also the one selected while viewing/editing
* the product afterward.
*
* Set once per import run from the operator's own answer (see
* Command\MigrateImportCommand::askImportLocale(), threaded through
* ImportSpec::$locale) at the top of Importer::import() — every resolver
* downstream (ProductAttributeResolver, ProductOptionResolver,
* CollectionResolver, ImportAttributeResolver) calls code() exactly as
* before, unaware anything changed. Falls back to Language::getDefault()
* only when nothing was ever set (e.g. an import path with no locale
* concept of its own — JudgeMe's reviews-only export never calls set()).
*/
class ImportLocale
{
private static ?string $code = null;
public static function set(string $code): void
{
self::$code = $code;
}
public static function code(): string
{
return self::$code ?? Language::getDefault()->code;
}
/**
* A static property outlives a single request only inside a
* long-running worker process, where a queued job for one import
* must not leak its locale into the next — called at the end of
* every Importer::import() run, success or failure.
*/
public static function reset(): void
{
self::$code = null;
}
}
@@ -1,10 +1,12 @@
<?php <?php
namespace Modules\Core\MigrateImport; namespace Modules\Core\MigrateImport\Services;
use InvalidArgumentException; use InvalidArgumentException;
use Modules\Core\MigrateImport\JudgeMe\JudgeMeExportImporter; use Modules\Core\MigrateImport\DTOs\ImportSpec;
use Modules\Core\MigrateImport\Shopify\ShopifyExportImporter; use Modules\Core\MigrateImport\Contracts\Importer;
use Modules\Core\MigrateImport\JudgeMe\Services\JudgeMeExportImporter;
use Modules\Core\MigrateImport\Shopify\Services\ShopifyExportImporter;
class ImporterFactory class ImporterFactory
{ {
@@ -1,6 +1,6 @@
<?php <?php
namespace Modules\Core\MigrateImport\Shopify; namespace Modules\Core\MigrateImport\Shopify\DTOs;
class ProductGroup class ProductGroup
{ {
@@ -0,0 +1,65 @@
<?php
namespace Modules\Core\MigrateImport\Shopify\Jobs;
use Illuminate\Bus\Batchable;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Lunar\Models\CollectionGroup;
use Lunar\Models\Currency;
use Modules\Core\MigrateImport\Services\ImportLocale;
use Modules\Core\MigrateImport\Shopify\DTOs\ProductGroup;
use Modules\Core\MigrateImport\Shopify\Services\ShopifyExportImporter;
/**
* One product per job — ShopifyExportImporter::import() used to loop over
* every ProductGroup inline, inside RunMigrateImportJob's own single
* process. A large export (hundreds of products, each with variants,
* media downloads and conversions) grew that one process's memory past
* queue:work's --memory limit (see docker-compose.yml), which kills the
* worker mid-job; the container then restarts and the WHOLE import
* re-runs from row one, never actually finishing. Splitting into one
* job per product means memory resets between jobs (a fresh worker
* process picks up each one), and a restart only repeats whichever
* single product was in flight — ImportMapping's own per-handle
* resolve()/record() already makes re-importing the same product cheap
* and idempotent.
*/
class ImportShopifyProductJob implements ShouldQueue
{
use Batchable;
use Dispatchable;
use InteractsWithQueue;
use Queueable;
use SerializesModels;
public int $tries = 3;
public function __construct(
private readonly ProductGroup $group,
private readonly string $imagesPath,
private readonly CollectionGroup $collectionGroup,
private readonly Currency $currency,
private readonly ?string $locale,
) {
}
public function handle(ShopifyExportImporter $importer): void
{
if ($this->locale !== null) {
ImportLocale::set($this->locale);
}
try {
$importer->importProduct($this->group, $this->imagesPath, $this->collectionGroup, $this->currency);
} finally {
// A queue worker process outlives a single job — this must
// not leak into whichever product the same worker picks up
// next.
ImportLocale::reset();
}
}
}
@@ -7,7 +7,7 @@ use Lunar\FieldTypes\TranslatedText;
use Lunar\Models\Collection; use Lunar\Models\Collection;
use Lunar\Models\CollectionGroup; use Lunar\Models\CollectionGroup;
use Lunar\Models\Product; use Lunar\Models\Product;
use Modules\Core\MigrateImport\DefaultLocale; use Modules\Core\MigrateImport\Services\ImportLocale;
class CollectionResolver class CollectionResolver
{ {
@@ -45,7 +45,7 @@ class CollectionResolver
'collection_group_id' => $group->id, 'collection_group_id' => $group->id,
'attribute_data' => [ 'attribute_data' => [
'name' => new TranslatedText(collect([ 'name' => new TranslatedText(collect([
DefaultLocale::code() => new Text($name), ImportLocale::code() => new Text($name),
])), ])),
], ],
]); ]);
@@ -8,7 +8,7 @@ use Lunar\Models\Attribute;
use Lunar\Models\AttributeGroup; use Lunar\Models\AttributeGroup;
use Lunar\Models\Product; use Lunar\Models\Product;
use Lunar\Models\ProductType; use Lunar\Models\ProductType;
use Modules\Core\MigrateImport\DefaultLocale; use Modules\Core\MigrateImport\Services\ImportLocale;
class ImportAttributeResolver class ImportAttributeResolver
{ {
@@ -40,7 +40,7 @@ class ImportAttributeResolver
[ [
'attribute_group_id' => $group->id, 'attribute_group_id' => $group->id,
'position' => $nextPosition++, 'position' => $nextPosition++,
'name' => [DefaultLocale::code() => $definition['label']], 'name' => [ImportLocale::code() => $definition['label']],
'section' => 'main', 'section' => 'main',
'type' => $definition['type'], 'type' => $definition['type'],
'required' => false, 'required' => false,
@@ -49,7 +49,7 @@ class ImportAttributeResolver
? ['richtext' => false] ? ['richtext' => false]
: [], : [],
'system' => false, 'system' => false,
'description' => [DefaultLocale::code() => ''], 'description' => [ImportLocale::code() => ''],
], ],
); );
@@ -62,7 +62,7 @@ class ImportAttributeResolver
return AttributeGroup::firstOrCreate( return AttributeGroup::firstOrCreate(
['attributable_type' => Product::morphName(), 'handle' => 'import'], ['attributable_type' => Product::morphName(), 'handle' => 'import'],
[ [
'name' => [DefaultLocale::code() => 'Additional Details'], 'name' => [ImportLocale::code() => 'Additional Details'],
'position' => 100, 'position' => 100,
], ],
); );
@@ -6,7 +6,7 @@ use Lunar\FieldTypes\Number;
use Lunar\FieldTypes\Text; use Lunar\FieldTypes\Text;
use Lunar\FieldTypes\TranslatedText; use Lunar\FieldTypes\TranslatedText;
use Lunar\Models\ProductType; use Lunar\Models\ProductType;
use Modules\Core\MigrateImport\DefaultLocale; use Modules\Core\MigrateImport\Services\ImportLocale;
class ProductAttributeResolver class ProductAttributeResolver
{ {
@@ -43,7 +43,7 @@ class ProductAttributeResolver
} }
return new TranslatedText(collect([ return new TranslatedText(collect([
DefaultLocale::code() => new Text($value), ImportLocale::code() => new Text($value),
])); ]));
} }
} }
@@ -5,7 +5,7 @@ namespace Modules\Core\MigrateImport\Shopify\Resolvers;
use Illuminate\Support\Str; use Illuminate\Support\Str;
use Lunar\Models\ProductOption; use Lunar\Models\ProductOption;
use Lunar\Models\ProductOptionValue; use Lunar\Models\ProductOptionValue;
use Modules\Core\MigrateImport\DefaultLocale; use Modules\Core\MigrateImport\Services\ImportLocale;
class ProductOptionResolver class ProductOptionResolver
{ {
@@ -24,8 +24,8 @@ class ProductOptionResolver
return ProductOption::query()->firstOrCreate( return ProductOption::query()->firstOrCreate(
['handle' => $handle], ['handle' => $handle],
[ [
'name' => [DefaultLocale::code() => $name], 'name' => [ImportLocale::code() => $name],
'label' => [DefaultLocale::code() => $name], 'label' => [ImportLocale::code() => $name],
'shared' => true, 'shared' => true,
], ],
); );
@@ -46,7 +46,7 @@ class ProductOptionResolver
return $existing ?? ProductOptionValue::create([ return $existing ?? ProductOptionValue::create([
'product_option_id' => $option->id, 'product_option_id' => $option->id,
'name' => [DefaultLocale::code() => $value], 'name' => [ImportLocale::code() => $value],
]); ]);
} }
} }
@@ -1,8 +1,9 @@
<?php <?php
namespace Modules\Core\MigrateImport\Shopify; namespace Modules\Core\MigrateImport\Shopify\Services;
use RuntimeException; use RuntimeException;
use Modules\Core\MigrateImport\Shopify\DTOs\ProductGroup;
class ShopifyCsvReader class ShopifyCsvReader
{ {
@@ -1,9 +1,10 @@
<?php <?php
namespace Modules\Core\MigrateImport\Shopify; namespace Modules\Core\MigrateImport\Shopify\Services;
use Lunar\Models\TaxClass; use Lunar\Models\TaxClass;
use Lunar\Models\ProductOption; use Lunar\Models\ProductOption;
use Illuminate\Support\Facades\Bus;
use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Log;
use Lunar\Models\Collection; use Lunar\Models\Collection;
use Lunar\Models\CollectionGroup; use Lunar\Models\CollectionGroup;
@@ -12,9 +13,13 @@ use Lunar\Models\Language;
use Lunar\Models\Product; use Lunar\Models\Product;
use Lunar\Models\ProductVariant; use Lunar\Models\ProductVariant;
use Lunar\Models\Url; use Lunar\Models\Url;
use Modules\Core\MigrateImport\ImportSpec; use Modules\Core\Catalog\Services\SkuBackfillService;
use Modules\Core\MigrateImport\Importer; use Modules\Core\MigrateImport\Services\ImportLocale;
use Modules\Core\MigrateImport\DTOs\ImportSpec;
use Modules\Core\MigrateImport\Shopify\DTOs\ProductGroup;
use Modules\Core\MigrateImport\Contracts\Importer;
use Modules\Core\MigrateImport\Models\ImportMapping; use Modules\Core\MigrateImport\Models\ImportMapping;
use Modules\Core\MigrateImport\Shopify\Jobs\ImportShopifyProductJob;
use Modules\Core\MigrateImport\Shopify\Resolvers\AssetResolver; use Modules\Core\MigrateImport\Shopify\Resolvers\AssetResolver;
use Modules\Core\MigrateImport\Shopify\Resolvers\BrandResolver; use Modules\Core\MigrateImport\Shopify\Resolvers\BrandResolver;
use Modules\Core\MigrateImport\Shopify\Resolvers\CollectionResolver; use Modules\Core\MigrateImport\Shopify\Resolvers\CollectionResolver;
@@ -46,9 +51,21 @@ class ShopifyExportImporter implements Importer
) { ) {
} }
/**
* Dispatches one ImportShopifyProductJob per ProductGroup instead of
* importing them inline — see that job's own docblock for why (a
* single process holding every group in memory for the whole run
* kept exceeding queue:work's --memory limit on a large export,
* which kills the worker mid-run and restarts the entire import from
* scratch). Bus::batch()'s then() is what SkuBackfillService used to
* run right after this loop — now deferred until every product job
* in the batch has actually finished, since dispatching a batch
* itself returns immediately.
*/
public function import(ImportSpec $spec): void public function import(ImportSpec $spec): void
{ {
$groups = $this->csvReader->read($spec->filePath); $groups = $this->csvReader->read($spec->filePath);
$total = count($groups);
$imagesPath = dirname($spec->filePath).'/files'; $imagesPath = dirname($spec->filePath).'/files';
$collectionGroup = CollectionGroup::firstOrCreate( $collectionGroup = CollectionGroup::firstOrCreate(
['handle' => 'shopify'], ['handle' => 'shopify'],
@@ -56,12 +73,30 @@ class ShopifyExportImporter implements Importer
); );
$currency = Currency::getDefault(); $currency = Currency::getDefault();
foreach ($groups as $group) { Log::info('Shopify import: dispatching product jobs', ['total' => $total]);
$this->importProduct($group, $imagesPath, $collectionGroup, $currency);
} $jobs = collect($groups)->map(fn (ProductGroup $group) => new ImportShopifyProductJob(
$group,
$imagesPath,
$collectionGroup,
$currency,
$spec->locale,
))->all();
Bus::batch($jobs)
->name("Shopify import: {$spec->filePath}")
->then(function () use ($total) {
Log::info('Shopify import: all product jobs finished, backfilling missing SKUs', ['total' => $total]);
app(SkuBackfillService::class)->backfill();
Log::info('Shopify import: finished', ['total' => $total]);
})
->catch(function ($batch, $e) {
Log::error('Shopify import: batch failed', ['error' => $e->getMessage()]);
})
->dispatch();
} }
private function importProduct( public function importProduct(
ProductGroup $group, ProductGroup $group,
string $imagesPath, string $imagesPath,
CollectionGroup $collectionGroup, CollectionGroup $collectionGroup,
@@ -100,7 +135,12 @@ class ShopifyExportImporter implements Importer
[ [
'element_type' => $product->getMorphClass(), 'element_type' => $product->getMorphClass(),
'element_id' => $product->id, 'element_id' => $product->id,
'language_id' => Language::getDefault()->id, // ImportLocale, not Language::getDefault() — same
// reasoning as ProductAttributeResolver et al.: this
// product's handle/slug came from the export, written in
// whatever language the operator said the file is in,
// not necessarily this store's own default language.
'language_id' => Language::where('code', ImportLocale::code())->value('id'),
], ],
[ [
'slug' => $group->handle, 'slug' => $group->handle,